@lindsay.at

Prompted by a tweet by Simon Willison on Monday, I was intrigued to hear about the Twitter real-time streaming APIs. In spare moments this week, rather than surfing the web, I found myself looking at how to get a view on the API from within Python which was… not trivial. In fact, none of the standard libraries seemed to handle the API at all: every HTTP access library waited until the stream closed, which was potentially forever.

A little poking and prodding, and I knew Twisted was capable of doing it. However, that seemed too heavy-weight a solution for just hacking about. I discovered asyncore, and despite the fairly thin documentation and examples online, it seemed clear that it could help. By evening I had knocked together something that worked with Twitter’s basic authentication and the proxy at work, which pretty much meant that I had to create my own basic HTTP/1.0 client. Not rocket science — I guess I had done the same thing 14 years earlier in Perl4 — but it took some trial and error, and neither the asynchat library nor any other common libraries did anything to simplify putting together an HTTP request.

I posted things up on github just to get some feedback on whether what I was doing was hopelessly misguided and reinventing the wheel, and — other than hearing it confirmed that Twisted could do this easily — it turns out no one else was doing this in Python. Cool, because they’re nifty APIs. It seems Twitter had turned into quite the target this year, with both Facebook on one hand and Google on another looking to get in on the action. I think the simplicity, transparency, and speed of the API are brilliant responses to things like Google Wave. This is very easy to work with, will keep developers around, and is pregnant with possibilities.

Oh, the speed. The fact that there is no apparent latency — a message that I send through my desktop client will appear in the stream before the miniature posting window even closes — makes this tremendously satisfying to work with.

After returning to the project like a bad rash, aided by encouraging words, github followers, and a primal pleasure in seeing words from all around the world spontaneously crawl up my screen, I’ve got a good chunk of code. It’s not brilliantly engineered, but I really enjoyed the process of it, the way it grew organically while trying it with different applications, such as a twistori clone to track (and highlight) multiple keywords in the Twitter stream.

There’s also fixreplies.py, a read-only client that mines your Friends, Followers, Favorites, and conversations to find people to track all public tweets to and from. While I think Twitter did the right thing in limiting replies, this makes for an interesting adjunct to a main, traditional twitter client. You get the feel for a lot more conversations sliding by. As it’s only text in the terminal, it feels more ephemeral, with far less of a need to catch up on everything.

Mostly as an exercise to see if it could be done, I also turned that client into a double-clickable script in Mac OS X. Click, and it opens up a terminal window, asks you for some information on your account, what sorts of users interest you, and how to find up to two hundred users’ conversations to listen in on.

I do hope that Twitter makes this available for desktop clients and the general public, because it really expands the, er, Twitterverse, and opens up new possibilities in interacting with and getting data from Twitter.

As previously blogged, I attended the BBC's Mashed08 hack day. I explored some of the noteworthy hacks on the LOLCODE site, but it's time that I explain what I did with my 24 hours straight of hacking.

The number of APIs and virtual toys unleashed by the BBC at Mashed08 was a bit dizzying with all the choices. In the end, however, I had to go with an idea that had been rattling around in my head for the longest: a twitter bot based on the Echo Nest Recommend API. Twitter bots are nothing new, not terribly original, and not even all that feasible nowadays with the API limits, but it seemed quite a nice application of the EN Recommender.

What I spent a lot of time on was the ergonomics, context awareness, and giving the bot a memory, all in aid of getting maximal information from minimal effort from the (mobile) twitter user. The bot's name, 'recomme,' was designed to be easily keyable with a T9 keypad. I spent a fair amount of time on maximizing the amount of information in 140 characters.

Given a tweet with a bandname to @recomme, it responds with the closest recommended bands:

If you name one the BBC's (pop) music radio stations, then the bot is aware of the context at the time: it goes off and checks on what's currently playing:

If you messaged the bot too late, however, you can correct it, asking for the track immediately preceding:

If the recommendations are on target, you can ask for more of the same:

As you can see, there's a fair bit of state saved with each interaction with the bot, and it responds with as much information it can fit into the space allotted.

The fairly terse URLs that follow each set of recommendations are give each query some persistence, an easily accessible reminder of what was requested, capturing the context of the moment, and offering more verbose detail than can be captured in a 140 character message. The user's past queries (saveable, sharable) are also accessible via a linked user page.

The cherry on top, and perhaps the only part of the hack that couldn't conceivably be done last year, was that the BBC Audio & Music Interactive team brought live archives of the BBC pop music radio stations. For the Mashed08 event, I was able to link to these live, time-indexed archives, so in the above "BBCR1" query, the persistent link pointed to the right time in the past such that you would hear the BeatFreakz song that was playing at that time.

Most of the development time was spent re-learning parts of Django, and getting the model for the underlying web application ("memory") right. The twitter bot is a separate process that requests information from and saves things to the Django webapp with some special POST requests.

The project was a a fair bit of fun, and it felt rewarding to see it through. I'm a bit conflicted on whether to deploy it: I think it would be good, fun, and useful to some, but it could easily use another solid 24 hours of polish before it's presentable. Furthermore, Twitter is in no state to support a new bot: 20 API requests per hour are nearly useless for something that's ostensibly an interactive mobile application.

As a postscript, I noticed that the "recomme.com" domain was cybersquatted by the time I returned to my hotel in London. I wasn't too bothered by it, as I have other domains that could be pressed into service for this, but I was mightily impressed at who's paying attention.