"Quadratic space will kill you faster than quadratic time"

Learnt it the hard way. Ouch.

(Prof Steven Skiena: Lecture on homology)

A distributed pipeline for processing text

Usually, Hadoop is the way to go.

However, I have joined a project that has been underway for more than a year, and the processes have been written in mostly an ad-hoc way - shell, python, and Java standalone programs. Converting each of these to mappers and reducers would have been an arduous task.

I decided to re-write the pipeline in SCons. There are many things about this pipeline that represent a conventional build. There are dependencies, and usually newer functionality/processing is added to the later stages of the pipeline. Luckily, SCons takes in regular python functions as "Builders", which I hooked into xml-rpc functions, and we soon had SCons running the pipeline on multiple servers (just five, actually - that's all we'd get for our pipeline). The file-system is an NFS share, which simplifies things a great deal.

Python, however, has been a bit on the slower side. Also, invoking the Java VM every time you need to process a file feels like too much of an overhead. So while the pipeline is functional, and processes the corpus much faster than before (5-6 hours vs 20+ earlier), we are considering re-writing the XML-RPC server in Java. The standalone programs can be easily ported to the server implementation, and invoking shell scripts from Java shouldn't be very different from invoking them from python - things should only improve. I wonder, however, if I should have written this in Hadoop to start with.

Correlation != Causation

(from PeteSearch)

My dog Thor hates getting wet, but even when there's rain lashing against the windows he still starts off dancing in circles when it's time for his walk. It's only when I pull out his yellow rain jacket that he slumps and stares at me mournfully. He seems convinced that if I just left the jacket off, the rain would go away.

Much as I try and convince him of the error in his logic, he's unmoved, and it's hard to blame him. Humans will happily swallow studies that use the weasel word 'link' to claim something that is associated with an outcome is its cause … I half-expect to get up one morning and discover that Thor's eaten the raincoat, in the hope of bringing back the sun …

Random funny

(from http://neil.fraser.name/news/2007/04/07/ ... the carving there is great too!)

Early Friday morning I walked by a garbage can outside one of the Google cafés. A bird which had been feeding on scraps saw me coming, exploded out of the garbage can and flew off. A passing coworker looked at me incredulously and said "Someone threw out a perfectly good bird."

The other side

This is my first attempt at making noodles, the Indo-Chinese way. I put a bit too much of soy sauce, but otherwise, it was nice. In fact, it was good enough for Hao to prefer it over the cucumbers that he had cooked! I wasn't that great with the chopsticks though, and used them as long as the volume of noodles was large enough for me to aim at.

Finally ...

... I had someone from China mistake me for a Chinese person. This, after years of being told that I look like Chinese person while in India. Inevitable, I guess.

Cabbage and rice tastes great!

Or maybe it is the hunger speaking. I didn't have breakfast - skipped directly to lunch.

This was a day after I visited NYC. Thanks to Shrikanth, I have some photos from my university now.

Have a look. NYC photographs will follow soon.

Edit: Let's try a slideshow -

The weaker sex?

From http://esr.ibiblio.org/?p=2118:

... women, in general, are not willing to eat the kind of shit that men will swallow to work in this field. Now let’s talk about death marches, mandatory uncompensated overtime, the beeper on the belt, and having no life. Men accept these conditions because they’re easily hooked into a monomaniacal, warrior-ethic way of thinking in which achievement of the mission is everything. Women, not so much. Much sooner than a man would, a woman will ask: "Why, exactly, am I putting up with this?"

... if we really want to fix the problem of too few women in computing, we need to ask some much harder questions about how the field treats everyone in it.

More like the wiser one.

Using unicode to annotate emails

I have been using unicode characters like ★, ✘, ✔, ⚠ to annotate my gmail labels. I used it today to mark the subject of one of my mails as a high-priority mail. Wonder if it'll work.

Using javascript to avoid the mouse and page scrolling

Here's the problem. It is not very easy to scroll a document when you're inside an input element. Arrow keys don't work, and Page Up/Page Down jump in big increments. What if you want to see just a few lines below the current element? Our clients hate to scroll. And they hate having to use the mouse. This just brings the two together.

FScroll is a JQuery plug-in which makes a page scroll to the currently focussed element, keeping it's position centered with respect to the document. This helps keep a bit of "context" around the currently focussed element - since it is centered, you can see a few elements both above and below the currently focussed element.

Here are the sources. And here's a page explaining it's usage in some detail. And oh, it does nested centering too. But it requires that the 'nesting' container have a css styling of position : relative (in the demo page, the div enclosing the table is positioned relative). This was not strictly necessary, but it made the coding a bit easier. If you can't live with the styling restriction, let me know. I'll try to do what I can.

You may report issues here.