About a year and a half ago, I wanted a metadata database (or metabase), where I could collect all my notes about songs I’ve listened to and pictures I’ve taken.

The idea is that the photo and its associated notes are both data (metadata is also data). It’s important not to loose any of them. The hard part – for which I don’t know of an existing solution – is how to save the associations.

For multiple reasons, it is not a good idea to specify the association of datum by placing them in the same file. If I had original and cropped versions of the same photograph, I wouldn’t like to “glue” the two versions together by placing them in the same file. Also, computer programs aren’t good at guessing which files are related to which, so placing several mp3 files together in the same directory may not automatically create a mixtape. (If you’ve ever used an iPod you know what I’m talking about.)

Thus, the original idea of a metabase is now not as much about storing metadata, as it is about storing associations.
Such a database would store pairs of associated GUIDs in one table, and assorted external identifiers (filenames, hashes) for each GUID in another.

Of course, somebody has already thought of this.

I’m one of the unlucky people living in far-away places that don’t get Twitter updates via SMS. Recently, Twitter activity has increased among some of my friends, so I wanted to get updates without having to remember to visit the site (I’m lazy that way).

Long story short, the “friends_timeline” RSS feed that twitter offers is password protected, whereas Google Reader doesn’t support passwords (and even if it did, I wouldn’t want to put my Twitter password there). So my idea was to create a non-password protected proxy page that takes care of entering the password and returning the contents of the feed. Below is a simple PHP script that does just that. You can put it on your web server and subscribe to it using your choice of RSS reader, no passwords required.

<?php
// A simple proxy to fetch password-protected pages on services that don't
// support it. Example: twitter feed on google reader
//
// based on:
// http://developer.yahoo.com/javascript/samples/proxy/php_proxy_simple.txt
//
// Author: Jason Levitt
// December 7th, 2005
//
// Modifications: Ehud Meiri, Feb 2009

$url = 'http://USERNAME:PASSWORD@twitter.com/statuses/friends_timeline/NUMBER.rss';

// Open the Curl session
$session = curl_init($url);

// Don't return HTTP headers. Do return the contents of the call
curl_setopt($session, CURLOPT_HEADER, false);
curl_setopt($session, CURLOPT_RETURNTRANSFER, true);

// Make the call
$xml = curl_exec($session);

echo $xml;
curl_close($session);

?>

To use it for your account, change the “$url” line to match your details (you can find your NUMBER in the “RSS feed” link here).

Note that this solution is insecure (esp. on shared machines), as the password is stored in plaintext in the script and anybody who can read the script source can know your password. Be careful.

I’ve just purchased a new toy: the Wacom Bamboo tablet. Without any modifications, the tablet behaves in X like a (very slow) touchpad.
Luckily, kernel and xorg drivers are available, though support is not mature yet (I was expecting some gnome configuration tool).

Installing a Wacom Bamboo on Debian testing:
A. Install the following packages:
Packages that should be already on your system:
1. xserver-xorg-input-wacom (version 0.7.9.3-2)
2. linux-image-2.6.24-1-686 (or possibly 2.6.23, 2.6.22, YMMV)

Packages that you probably don’t have yet:
3. wacom-tools (version 0.7.9.3-2)

B. Tweak your /etc/X11/xorg.conf:
1. Add the following sections.

Section "InputDevice"
        Identifier      "stylus"
        Driver          "wacom"
        Option          "Device"                "/dev/input/wacom"
        Option          "Type"                  "stylus"
        Option          "USB"                   "on"
EndSection

Section "InputDevice"
        Identifier      "eraser"
        Driver          "wacom"
        Option          "Device"                "/dev/input/wacom"
        Option          "Type"                  "eraser"
        Option          "USB"                   "on"
EndSection

Section "InputDevice"
        Identifier      "cursor"
        Driver          "wacom"
        Option          "Device"                "/dev/input/wacom"
        Option          "Type"                  "cursor"
        Option          "USB"                   "on"
EndSection

Section "InputDevice"
        Identifier      "pad"
        Driver          "wacom"
        Option          "Device"                "/dev/input/wacom"
        Option          "Type"                  "pad"
        Option          "USB"                   "on"
EndSection

2. Change the following section to look something like this:

Section "ServerLayout"
        Identifier      "Default Layout"
        Screen          "Default Screen"
        InputDevice     "Generic Keyboard"
        InputDevice     "Configured Mouse"
        InputDevice     "stylus"                "SendCoreEvents"
        InputDevice     "eraser"                "SendCoreEvents"
        InputDevice     "cursor"                "SendCoreEvents"
        InputDevice     "pad"                   # no core events
EndSection

C. Configure the touch ring to act like a mouse wheel (i.e. scrolling).
1. I put the configuration in the in file /etc/X11/Xsession.d/80wacom-config_scrollpad, for lack of a better place.
Its contents:

xsetwacom set pad AbsWDn "5"
xsetwacom set pad AbsWUp "4"

D. That’s it.

More documentation and options here.

Robert X. Cringley writes:

We’re talking about tin whiskers, single crystals that mysteriously grow from pure tin joints but not generally from tin-lead solder joints. Nobody knows how or why these whiskers grow and nobody knows how to stop them, except through the use of lead solder. Whiskers can start growing in a decade or a year or a day after manufacture. They can grow at up to nine millimeters per year. They grow in any atmosphere including a pure vacuum. They grow in any humidity condition. They just grow. And when they get long enough they either touch another joint, shorting out one or more connections, or they vaporize in a flash, creating a little plasma cloud that can carry for an instant hundreds of amps and literally blow your device to pieces.

TMB (Too Many Bookmarks) is my homepage script. It is intended to give quick access to my most frequently used bookmarks, from any computer I may be using. I used to have a wiki page that did this, but now TMB allows me to easily add bookmarks, assign their category, and (roughly) order them.
It is very very basic, but it works, it’s fast, and I love it.

Requirements: PHP 5.2 with SQLite 3

Download it at the TMB page.

BTW Instapaper is a great tool for bookmarking pages that you want to read later. You can flag pages as read/unread, add pages to your list with a press of a button (I might add this to TMB), and the registration is really simple.

We used to write important information on the back of photographs. Today they are digital, but we would still like to remember, years from now, where who and what is in those photos we took.

Wrestling the information into the filename might be a good idea. It assures us that the metadata (descriptive info) is copied along with the data (the file). However, it’s not an elegant solution for most common filesystems, since they place restrictions on the length of filenames and character sets used. Ideally, a block of descriptive metadata should replace filenames altogether.

Most modern file formats include an internal metadata format (JPEG has Exif, MP3 has ID3), and maybe the solution is to use several metadata implementations (de facto standards). But what I need is to be able to take a piece of text, an image, or a web page snapshot, and associate it to an arbitrary file. I also want the associated data to accessible from the context of that file, be it that of a text viewer, media player, or file manager.

Is there a metadata standard, or is that just an oxymoron? I mean, metadata can be, by definition, all types of data. So writing such a standard would mean to anticipate all future kinds of data.

Is metadata necessary? You can argue that there is no end to metadata and that it’s pointless. Yet we still find ourselves appending notes to images, tagging cats, and blogging about other people’s creations.

The Room (imdb) is a movie in the “so bad it’s good” category. Not having seen it yet, just reading the reviews in imdb and seeing the clips on Mahalo Daily got me giddy with excitement. I love how the interviewer managed to keep a straight face throughout the interview, except at the very end.

Comment excerpts:

So Independent it’s Cinematic Masturbation!
– Mroussele

The centerpiece of this filmic holocaust is Mr. Tommy Wiseau himself. Without him, it would still be the worst movie ever made, but with him it is the greatest worst movie ever made. Tommy has been described as a Cajun, a Croatian cyborg, possibly from Belgium, clearly a product of Denmark, or maybe even not from this world or dimension. All of these things are true at any one moment. He is a tantalizing mystery stuffed inside an enigma wrapped in bacon and smothered in cheese.
– Brickyard Jimmy

[Illustration credit Andrice Arp, courtesy of BookMooch.com]

Regarding my previous post about Twitter’s database architecture and scaling, here’s a post about bookmooch.com’s scaling troubles (bookmooch is a book swapping site).

Bookmooch had a search engine for books, using a single table that indexed each word in a book’s title/author/etc. This system was very efficient for searching (one query per word). The problem was that adding new books to the index was very costly (the more popular the word was, the longer it took to update). It was up to the point where the disk writes took as much as 20 seconds out of their alloted minute.

In short, the solution was to have two tables for the index instead of one. One table was optimized for queries (faster searches), while the other for updates (adding new books quickly). These tables would be synchronized periodically.

IMHO, the guy in charge did nothing wrong with the initial design of the search engine. There are certain characteristics of a system that are hard to predict and even then, one needs a lot of experience to correctly optimize things in advance (like the problem in the second paragraph). As in the Twitter story, profiling saved the day, but don’t forget to monitor your logs!

Out of boredom:

1. Name for a site (only Hebrew speakers will get this): vayehi.org
2. Name for a girl: lolcat

Dimethyltryptamine (DMT) is the alleged molecule that the brain produces during REM sleep that causes us to dream. Rogan claims in the video below that sleep is a psychedelic experience and that taking DMT allows you to experience that. One side-effect is that, just like regular dreams, you quickly forget the experience once it’s over.

Another day, another ordeal getting Linux to work in a Windows world. And after spending one trying to configure L2TP to work in Israel, I felt the need to collect the knowledge I gathered in one place, so that future souls may save some time.

Here’s a new search feature I found: glob patterns, e.g., using asterisks in place of words you don’t know.
Try this search: “one of these days alice * to the moon”
I found it works in both Google and Yahoo search, but not in Microsoft live search.

munroe from tinymixtapes doesn’t like Rafter’s Music for Total Chickens, and I, for one, agree:

But fuck my dignity. I’m making a stand: This record is bad. It’s uneven, sparse to a fault, weird for the sake of being weird, and generally not entertaining in the least. Every time Rafter starts moving toward something worth hearing, they back off, leaving me holding my dick. I don’t like that. When music starts to swell and build, so do I. If I don’t get some kind of sonic release, my nuts get grumpy. Nobody likes grumpy nuts.

Cringely claims that Adobe is making a move for the embedded and mobile device market.

Good GUIs are expensive and take time to perfect. Cringely says that it would save money for the companies involved if they could write their GUIs from just one platform and use Flash’s superior tools. It would also be nice if this would improve GUIs by making them cheaper, thereby saving us from bad design.

He has a point: each device may have a different ISA (Instruction Set Architecture) and/or OS. The ipods for instance use an ARM CPU by PortalPlayer running some custom OS (BTW rockbox rox!),
the iphone uses a different ARM chip running a version of OS X, and my Nokia N73 uses yet another ARM chip running the Symbian OS.
The same goes for other embedded devices such as cash registers, ATMs and other kiosks, and many of your future electronic appliances (most microwave ovens’ digital interfaces suck).

The funny thing is, that Adobe only has to promise cross-platform compatibility, without actually having to port Flash/whatever to every single platform (open source implementations may fill this niche). People just need to feel safe in order to commit, and Flash has a good track record. (Flash is backwards-compatible and compiled “animations” play exactly the same on every platform.)
The software market for mobile devices can’t really take off until they do.

My first project, which I will be starting in about a week, will be a web-based RSS/ATOM feed aggregator with the following properties:

1. It will store a complete archive of every item it sees.
2. Queries of the first ~50 items of each feed will be fast and take constant time.
3. The interface will be responsive as possible, despite server/communication latency.

The most important question thus far is whether to use a fully-fledged database with server (mysql, postgres, etc.), or to use a file based database (Berkeley db, etc.). The crucial factor is the database’s ability to achieve both properties 1 & 2 above. In other words, how well does each approach scale?

I believe that the second property could be achieved by using a table of all feed items sorted by their dates and limiting the first query to the first 20 items, fetching the next 20 items whenever the user reaches the end of the list (a la Google Reader).

The third property may be implemented using asynchronous commands to the server. Since the javascript code will be mostly event driven, I don’t think it’d be a problem making it perform several simultaneous actions (server requests) at once, provided they don’t conflict (e.g., access the same data structures).
Another way is to use a queue of commands waiting to be issued to the server. This has the added benefit of being able to reissue any failed commands at a later time, and being able to work offline.

Here’s an article that touches on the effect of heightened social awareness that twitter brings (like the awareness of the location of one’s limbs).

I thought that it would be automatic and all, but it wasn’t. The most frustrating thing was trying to find any official documentation (howtos, faqs, etc.). Most of the existing pages haven’t been updated since 2003, but the biggest problem is the number of drivers and packages developed over the years (linux-wlan, linux wireless extensions, ndiswrapper), that hindered IMO the development of good high-level configuration packages that configure, enable, and most importantly, troubleshoot the connection.

After a LOT of trial and error, and using this very helpful Ubuntu Forums post, I found the following settings in /etc/network/interfaces allow me to connect the 3com 3crusb10075 usb wifi dongle (with the zd1121 chipset) to a Linksys AP using WPA2 with the kernel drivers provided. It wasn’t at all obvious that I had to tell wpa-supplicant to enable both WPA and WPA2, as the forum posting called it, but then again it probably appears somewhere in one of the 67(!) pages of replies to that post.

My /etc/network/interfaces:

auto eth3
iface eth3 inet dhcp
wpa-ssid [essid]
wpa-ap-scan 1
wpa-proto WPA RSN
wpa-pairwise TKIP CCMP
wpa-group TKIP CCMP
wpa-key-mgmt WPA-PSK
wpa-psk [secret hex thing generated using wpa_passphrase]


Packages I used: wpasupplicant (0.6.0-1), zd1211-firmware (2.16.0.0-0.1), linux-image-2.6.21-2-k7 (2.6.21-5)

Rolling on Rails: Under the Hood at Twitter – talks about the “scramble to scale” and the importance of thought out database schemas and queries.

In order to scale your service, it is usually not necessary to change the entire framework. Profiling your code and queries can lead to small incremental changes that solve major bottlenecks (caches, query optimization, etc.).