@lindsay.at

I am a patient person, so it's only now that Twitter's perennial scaling problems are bothering me me enough to blog about them. That probably makes me the last person to do so, ever.

However, lately, it's hurt. It hurt most when I tried to implement a twitter bot at Mashed08. With API calls throttled down to 20 per hour, the best I could hope to do (via polling, and with IM shut out, that was the only obvious path) was to be a bot for one person making no more than one request every 10 minutes. So for the demo, the twitter connection was really baling wire and duct tape (or, ipython console and cut-and-paste into twitter's web form).

Last month, I read Tim Bray's Twitterbucks entry with interest. When I last checked in, nobody seemed to be interested in where the real scaling problems were, so the comment thread didn't come up with any real revelations.

Today, as I tried to reflect on why I use twitter, I came upon another potential solution: pay for what is the hardest to scale: disk access. When any high-volume application has to hit the spindles, it takes a massive performance hit. Twitter's recent outages seem to address that at least partially: paging backwards in your personal + friends timeline is scaled back, as are examining replies.

Seems to me that much of what Twitter covers well is the "now" and recent past. Going back in time on a merged timeline makes for increasingly expensive queries, and reaching further back in history goes beyond the memory caches. If Twitter didn't try to keep all its posts accessible, it could be a much more efficient messaging platform, always living in an amnesiac present. By having a web-accessible memory, with persistent tweets, it becomes a lot more difficult to predict where the database is going to be hit.

So take a look at Twitter as it stands right now. With the buttons that are disabled, which ones are the biggest pains? Single-user pagination? Friend pagination? Replies? Seems to me that the biggest omission is in having zero reply-page functionality, but the complex query (user > friends > friends' updates that can be seen > merged and sorted in time) database hit makes sense to limit. Why not cull functionality for all users such that it's either a complex query that hits memcached exclusively (the pages representing what's happening now and in the recent past), or a very trivial query that is allowed to hit the disks (a single, permalinked tweet or any user's front page of recent tweets). A twitter caught in the present, and exhibiting some memory when specifically prodded.

From there, you could charge for more archival access. I imagine this not as a monetisation move, or even one that could directly cover additional costs, but one that would allow serious users with serious needs self-select, not unlike what Flickr has done with their paid accounts. A paid user could access their archives as a continual stream of tweets on a blog-like page. They could access a more comprehensive memory of their friends' replies. They might even be given persistent past per-day or per-month archive pages.

I'll admit that I don't fully appreciate the particular scaling problems presented by heavy users like Scoble. Perhaps there are payment thresholds to pass once you follow 500 and 5000 users?

What do people think? I know I can't be the first to suggest it, but it's the first I've heard of it.

As previously blogged, I attended the BBC's Mashed08 hack day. I explored some of the noteworthy hacks on the LOLCODE site, but it's time that I explain what I did with my 24 hours straight of hacking.

The number of APIs and virtual toys unleashed by the BBC at Mashed08 was a bit dizzying with all the choices. In the end, however, I had to go with an idea that had been rattling around in my head for the longest: a twitter bot based on the Echo Nest Recommend API. Twitter bots are nothing new, not terribly original, and not even all that feasible nowadays with the API limits, but it seemed quite a nice application of the EN Recommender.

What I spent a lot of time on was the ergonomics, context awareness, and giving the bot a memory, all in aid of getting maximal information from minimal effort from the (mobile) twitter user. The bot's name, 'recomme,' was designed to be easily keyable with a T9 keypad. I spent a fair amount of time on maximizing the amount of information in 140 characters.

Given a tweet with a bandname to @recomme, it responds with the closest recommended bands:

If you name one the BBC's (pop) music radio stations, then the bot is aware of the context at the time: it goes off and checks on what's currently playing:

If you messaged the bot too late, however, you can correct it, asking for the track immediately preceding:

If the recommendations are on target, you can ask for more of the same:

As you can see, there's a fair bit of state saved with each interaction with the bot, and it responds with as much information it can fit into the space allotted.

The fairly terse URLs that follow each set of recommendations are give each query some persistence, an easily accessible reminder of what was requested, capturing the context of the moment, and offering more verbose detail than can be captured in a 140 character message. The user's past queries (saveable, sharable) are also accessible via a linked user page.

The cherry on top, and perhaps the only part of the hack that couldn't conceivably be done last year, was that the BBC Audio & Music Interactive team brought live archives of the BBC pop music radio stations. For the Mashed08 event, I was able to link to these live, time-indexed archives, so in the above "BBCR1" query, the persistent link pointed to the right time in the past such that you would hear the BeatFreakz song that was playing at that time.

Most of the development time was spent re-learning parts of Django, and getting the model for the underlying web application ("memory") right. The twitter bot is a separate process that requests information from and saves things to the Django webapp with some special POST requests.

The project was a a fair bit of fun, and it felt rewarding to see it through. I'm a bit conflicted on whether to deploy it: I think it would be good, fun, and useful to some, but it could easily use another solid 24 hours of polish before it's presentable. Furthermore, Twitter is in no state to support a new bot: 20 API requests per hour are nearly useless for something that's ostensibly an interactive mobile application.

As a postscript, I noticed that the "recomme.com" domain was cybersquatted by the time I returned to my hotel in London. I wasn't too bothered by it, as I have other domains that could be pressed into service for this, but I was mightily impressed at who's paying attention.

Last month I was requested to reflect on an aspect of my experience at ROFLCon. I did so, but forgot to cross-post what I wrote here. Augmented Reality Overload: A Digital Natives Guest Post:

Last week, in my post on ROFLCon, I wrote that

All barriers between real life and digital life seemed to collapse during the conference. In an audience full of laptops, iPhones, BlackBerries, and digital cameras, the volume of instant commentary created was enormous and baffling. But it wasn’t even just the audience: Adam Lindsay [a panel member, representing his website LOLCode] apparently live-posted much of his own panel directly to Twitter, between answering questions about LOLCode. And he wasn’t alone.

Only problem: I was wrong. In the comments, Adam Lindsay kindly noted that while he had planned to Twitter from the panel, and had in fact mentioned that he might (hence my confusion), in the end he wasn’t able to divide his attention that many ways at once. After reading this comment, it struck me that it might be valuable to see to what extent a digitally adept adult— in a sea of other digitally adept adults—experienced a digital usage generation gap between himself and his peers vs. the hundreds of college students milling about at ROFLCon. Adam was kind enough to oblige with the following guest post.

It’s worth noting that Adam’s views are his own, and though I’ve chosen to reprint them here in their entirety, they should be taken as one personal narrative rather than representative of any official position of the Digital Natives project. That said, we’re incredibly grateful to Adam for sharing his acute observations and personal experience, for writing such a thought-provoking piece, and for being willing to enter into this discussion in the first place. We look forward to hearing your responses.

-Diana Kimball, Digital Natives intern

I am 35 years old. I have been on the internet for half of my life. I “speak digital” fluently, but I have to admit that I don’t speak it as a native, and likely never will. I’m a dinosaur in this world, and I know I will be replaced by more nimble successors.

I’ve been thinking about it, and my big failing is that I can’t multitask like the younger set, or as women are famed for doing. The cost of switching between digital life and real life has a real mental cost, especially in terms of attention, in the psychological sense. And the reason why I feel as though I am not a digital native is that I feel this cost with every transition. I’m monolithic. I can think about one thing at a time. I am Windows 3.11. I am Mac OS 7.

This hit home for me during ROFLCon, where, armed with iPod touch and a long-established Twitter account, I thought I could keep up with the real and virtual worlds into which I was immersed. My droll plan was to live-twitter during my own panel, perhaps with just a “HAI WORLD. IM IN UR PANEL…” As it was, the panel was too fast-paced, and required too much of my attention just to keep on top of it. I had to be present, in the moment, and any step away from that would have been unfair to everyone else in the room.

I would try to keep an eye on Twitter during the conference while I was in the audience, but always ended up discovering that focusing on it took too much away from my understanding and enjoyment of what was going on in the panels. Twitter was useful, it did enhance the experience, and I believe it was the right medium that consensus established as a backchannel. However, for me, it was a matter of stepping out of the real life experience of the Con, and stepping into a digital reality. It was not an augmented reality; for me, ROFLCon’s digital backchannel was simply another parallel session track that I could follow. It seemed clear that for the younger set at ROFLCon, there was far less difficulty in meshing the real with the virtual.

So. I don’t feel like I’m a digital native. There are some trends, some things my younger colleagues love, that I know I don’t share in fully. I think they point to my failure to change context nimbly, but it could possibly just be me. What do you think?

Unlike nearly every person out there, I hate watching video while at the computer keyboard (with one notable exception). That is, I hate browsing through YouTube. Thankfully, I don’t have to miss out on videos of cute kittens, theremins, or both together, because I can defer YouTube to its proper context: I mark a promising video as a “favorite,” and when the mood strikes me, I use my AppleTV to catch up on all these effectively bookmarked videos while lying back on my couch.

I love music. I have spent tens of thousands of dollars (erk) on a legitimate music collection over the years. Yet I hardly listen to it, primarily because I can’t listen to most music whilst working. Music with lyrics (and I love intelligent lyricists) just collides with all the reading and writing I’m constantly doing at the keyboard.

As I close in on finishing a large, overdue project, I realize more and more that I need to swear off instant messaging. I have been instant messaging, in one form or another, ever since 1990, when I stepped onto the proper internet. I like to think I communicate well in the medium. However, IM takes far too much of my attention: I dedicate myself far too intensely to a real-time conversation, and everything else that I try to do is disrupted. I don’t dip into chats while continuing with my emailing, my writing, my editing, my coding, or my web surfing. The chats eat my attention.

I look at the next generation, say, my three-year-old daughter, and am in awe of what they will accomplish, having all the collected knowledge of the world wide web at their fingertips from day one. I’m am absolutely confident my daughter will develop the mental agility that I lack in order to mediate between the real and multiplicity of digital worlds she will be presented with. I hope, both as a parent and as an educator, I can help develop the critical facilities she and her peers will need to filter through all that information.

In the meantime, I’m having fun in this new world, myself. I get by pretty well with the language, after all. I know there will come a time when I will reach my limit, when all I can do is sit back and watch in wonderment what these kids can do.

But not just yet.

Adam Lindsay finally got used to being called The LOLCODE Guy while at ROFLCon. He sporadically blogs at http://lindsay.at/blog/, and formerly blogged more regularly on the role of digital multimedia tools for research in the humanities at http://mediadescri.be.

(Originally at Digital Natives.)

Bewitched on iTunes Store UK - detail, originally uploaded by a_t_l.

I had never noticed differential pricing within a normal sitcom series before. In the UK, £1.89 (~3.70USD) is the nominal price for a TV Programme. With certain "catalogue" titles, they have been recently experimenting with £1.19 (~2.33USD) per episode. This is the first place I've seen both next to one another.

It might make sense for Apple and partners to experiment with increased differential pricing abroad, first, before launching it in the US. The partners are happy because either price is still a premium compared with US prices.

On the other hand, it could be a mistake, in view of there being two episode number sixes.

phobos.apple.com/WebObjects/MZStore.woa/wa/viewTVSeason?i...

So, after weeks of dithering, talking with my colleagues and hearing their enthusiasm for the event provided me enough impetus to attend BBC's Mashed. It's HackDay 2008, basically, and it really felt like I missed something special last year. I've been getting my hack-fu together, and it felt like it would be nifty to go as a team.

But no, said colleagues dropped out almost immediately.

Mother-in-law came to the rescue, though, in offering to take Helena for the weekend, un-prompted. Great! (What? She's now expressing mixed feelings?…) So, I'm committed to going, even if I'm not going to approach it as the young man's game it appears to be. I have a hotel, I have a train, and I will attempt to treat myself right in the process. I want to hack, but I'll be fairly happy soaking up the atmosphere and giving out most of my remaining LOLCODE stickers, if people are interested.

For the rest: we shall see what we shall see.

Véro Pepperrell posted at her blog on the new iPhone announcement:

However, the biggest change isn’t in the physical device. It’s all in the perception. Last time around, Apple was looking for early adopters, geeks and IWOOTs* to test-run their product in a giant, live usability testing session. Now that they’ve been able to watch us use the device, it’s time to reach out to the normobs with lower upfront costs.

[* Def. IWOOT: “I want one of those”, otherwise known as saddos like me who can’t resist the latest gadget, even at exorbitant prices.]

Back in the day of frequent IM conversation with Fraser Speirs, we jointly coined a term regarding price points, OGOT. It stands for the rather British, "Oh, Go On, Then," and here signifying a price point that is perceived as being low enough for an impulse purchase. It's low enough to break through any price-based purchase resistance. It varies per product or per person, but you know it when you see it.

That's been the major focus of Apple's revamped iPhone, to address the reasons for purchase resistance head-on. The biggest reason that affects the most people has been price. In the UK, £99 sounds absolutely reasonable for a full-featured mobile on a £30-35/month plan. Oh, go on, then.

The iPhone steps out of the realm of the élite device into the mass market. Fantastic!