Archive for June, 2007

Project: RSS

My first project, which I will be starting in about a week, will be a web-based RSS/ATOM feed aggregator with the following properties:

  1. It will store a complete archive of every item it sees.
  2. Queries of the first ~50 items of each feed will be fast and take constant time.
  3. The interface will be responsive as possible, despite server/communication latency.

The most important question thus far is whether to use a fully-fledged database with server (mysql, postgres, etc.), or to use a file based database (Berkeley db, etc.). The crucial factor is the database’s ability to achieve both properties 1 & 2 above. In other words, how well does each approach scale?

I believe that the second property could be achieved by using a table of all feed items sorted by their dates and limiting the first query to the first 20 items, fetching the next 20 items whenever the user reaches the end of the list (a la Google Reader).

The third property may be implemented using asynchronous commands to the server. Since the javascript code will be mostly event driven, I don’t think it’d be a problem making it perform several simultaneous actions (server requests) at once, provided they don’t conflict (e.g., access the same data structures).
Another way is to use a queue of commands waiting to be issued to the server. This has the added benefit of being able to reissue any failed commands at a later time, and being able to work offline.

Read Full Post »

More about Twitter

Here’s an article that touches on the effect of heightened social awareness that twitter brings (like the awareness of the location of one’s limbs).

Read Full Post »

I thought that it would be automatic and all, but it wasn’t. The most frustrating thing was trying to find any official documentation (howtos, faqs, etc.). Most of the existing pages haven’t been updated since 2003, but the biggest problem is the number of drivers and packages developed over the years (linux-wlan, linux wireless extensions, ndiswrapper), that hindered IMO the development of good high-level configuration packages that configure, enable, and most importantly, troubleshoot the connection.

After a LOT of trial and error, and using this very helpful Ubuntu Forums post, I found the following settings in /etc/network/interfaces allow me to connect the 3com 3crusb10075 usb wifi dongle (with the zd1121 chipset) to a Linksys AP using WPA2 with the kernel drivers provided. It wasn’t at all obvious that I had to tell wpa-supplicant to enable both WPA and WPA2, as the forum posting called it, but then again it probably appears somewhere in one of the 67(!) pages of replies to that post.

My /etc/network/interfaces:

auto eth3
iface eth3 inet dhcp
wpa-ssid [essid]
wpa-ap-scan 1
wpa-proto WPA RSN
wpa-pairwise TKIP CCMP
wpa-group TKIP CCMP
wpa-key-mgmt WPA-PSK
wpa-psk [secret hex thing generated using wpa_passphrase]

Packages I used: wpasupplicant (0.6.0-1), zd1211-firmware (, linux-image-2.6.21-2-k7 (2.6.21-5)

Read Full Post »

Twitter’s architecture

Rolling on Rails: Under the Hood at Twitter – talks about the “scramble to scale” and the importance of thought out database schemas and queries.

In order to scale your service, it is usually not necessary to change the entire framework. Profiling your code and queries can lead to small incremental changes that solve major bottlenecks (caches, query optimization, etc.).

Read Full Post »