An Associative Database

About a year and a half ago, I wanted a metadata database (or metabase), where I could collect all my notes about songs I’ve listened to and pictures I’ve taken.
The idea is that the photo and its associated notes are both data (metadata is also data). It’s important not to loose any of them. The [...]

Read Full Post »

Bookmooch’s Search Engine

[Illustration credit Andrice Arp, courtesy of BookMooch.com]
Regarding my previous post about Twitter’s database architecture and scaling, here’s a post about bookmooch.com’s scaling troubles (bookmooch is a book swapping site).
Bookmooch had a search engine for books, using a single table that indexed each word in a book’s title/author/etc. This system was very efficient for searching (one [...]

Read Full Post »

Project: RSS

My first project, which I will be starting in about a week, will be a web-based RSS/ATOM feed aggregator with the following properties:

It will store a complete archive of every item it sees.
Queries of the first ~50 items of each feed will be fast and take constant time.
The interface will be responsive as possible, despite [...]

Read Full Post »

Twitter’s architecture

Rolling on Rails: Under the Hood at Twitter – talks about the “scramble to scale” and the importance of thought out database schemas and queries.
In order to scale your service, it is usually not necessary to change the entire framework. Profiling your code and queries can lead to small incremental changes that solve major bottlenecks [...]

Read Full Post »