@lindsay.at

I am a patient person, so it's only now that Twitter's perennial scaling problems are bothering me me enough to blog about them. That probably makes me the last person to do so, ever.

However, lately, it's hurt. It hurt most when I tried to implement a twitter bot at Mashed08. With API calls throttled down to 20 per hour, the best I could hope to do (via polling, and with IM shut out, that was the only obvious path) was to be a bot for one person making no more than one request every 10 minutes. So for the demo, the twitter connection was really baling wire and duct tape (or, ipython console and cut-and-paste into twitter's web form).

Last month, I read Tim Bray's Twitterbucks entry with interest. When I last checked in, nobody seemed to be interested in where the real scaling problems were, so the comment thread didn't come up with any real revelations.

Today, as I tried to reflect on why I use twitter, I came upon another potential solution: pay for what is the hardest to scale: disk access. When any high-volume application has to hit the spindles, it takes a massive performance hit. Twitter's recent outages seem to address that at least partially: paging backwards in your personal + friends timeline is scaled back, as are examining replies.

Seems to me that much of what Twitter covers well is the "now" and recent past. Going back in time on a merged timeline makes for increasingly expensive queries, and reaching further back in history goes beyond the memory caches. If Twitter didn't try to keep all its posts accessible, it could be a much more efficient messaging platform, always living in an amnesiac present. By having a web-accessible memory, with persistent tweets, it becomes a lot more difficult to predict where the database is going to be hit.

So take a look at Twitter as it stands right now. With the buttons that are disabled, which ones are the biggest pains? Single-user pagination? Friend pagination? Replies? Seems to me that the biggest omission is in having zero reply-page functionality, but the complex query (user > friends > friends' updates that can be seen > merged and sorted in time) database hit makes sense to limit. Why not cull functionality for all users such that it's either a complex query that hits memcached exclusively (the pages representing what's happening now and in the recent past), or a very trivial query that is allowed to hit the disks (a single, permalinked tweet or any user's front page of recent tweets). A twitter caught in the present, and exhibiting some memory when specifically prodded.

From there, you could charge for more archival access. I imagine this not as a monetisation move, or even one that could directly cover additional costs, but one that would allow serious users with serious needs self-select, not unlike what Flickr has done with their paid accounts. A paid user could access their archives as a continual stream of tweets on a blog-like page. They could access a more comprehensive memory of their friends' replies. They might even be given persistent past per-day or per-month archive pages.

I'll admit that I don't fully appreciate the particular scaling problems presented by heavy users like Scoble. Perhaps there are payment thresholds to pass once you follow 500 and 5000 users?

What do people think? I know I can't be the first to suggest it, but it's the first I've heard of it.

Bewitched on iTunes Store UK - detail, originally uploaded by a_t_l.

I had never noticed differential pricing within a normal sitcom series before. In the UK, £1.89 (~3.70USD) is the nominal price for a TV Programme. With certain "catalogue" titles, they have been recently experimenting with £1.19 (~2.33USD) per episode. This is the first place I've seen both next to one another.

It might make sense for Apple and partners to experiment with increased differential pricing abroad, first, before launching it in the US. The partners are happy because either price is still a premium compared with US prices.

On the other hand, it could be a mistake, in view of there being two episode number sixes.

phobos.apple.com/WebObjects/MZStore.woa/wa/viewTVSeason?i...

Véro Pepperrell posted at her blog on the new iPhone announcement:

However, the biggest change isn’t in the physical device. It’s all in the perception. Last time around, Apple was looking for early adopters, geeks and IWOOTs* to test-run their product in a giant, live usability testing session. Now that they’ve been able to watch us use the device, it’s time to reach out to the normobs with lower upfront costs.

[* Def. IWOOT: “I want one of those”, otherwise known as saddos like me who can’t resist the latest gadget, even at exorbitant prices.]

Back in the day of frequent IM conversation with Fraser Speirs, we jointly coined a term regarding price points, OGOT. It stands for the rather British, "Oh, Go On, Then," and here signifying a price point that is perceived as being low enough for an impulse purchase. It's low enough to break through any price-based purchase resistance. It varies per product or per person, but you know it when you see it.

That's been the major focus of Apple's revamped iPhone, to address the reasons for purchase resistance head-on. The biggest reason that affects the most people has been price. In the UK, £99 sounds absolutely reasonable for a full-featured mobile on a £30-35/month plan. Oh, go on, then.

The iPhone steps out of the realm of the élite device into the mass market. Fantastic!

Looks like HBO has put series like Deadwood, the Sopranos, and Sex in the City on the iTunes store. Good for them, and good for Apple. What I don't understand is how all the commentary points to how this "new" variable pricing for television episodes suddenly augurs a change in the dynamics between NBC and Apple.

I've watched these negotiations for a while, now. And it annoys me that it's widely reported that it's a cracking of the flat-rate pricing stance. TV shows have been priced at more than $1.99 for a while now. Look at PBS, look at Lifetime. It's premium content, and it still presents a compelling savings over the DVD pricing.

Now that I look at it, it seems like bub.bilicio.us and Doug Amoth of PC World got it right. I'm glad someone has.

Hey, has anyone else noticed that there has finally been a boost to the iTunes Plus library on Apple's iTunes store? I noticed while browsing the iTunes Wi-Fi music store, but confirmed it using the normal store.

There are some notable things about it:

• The independents have arrived: it's not a cluster of further EMI songs, but rather they appear to be from small labels.
• The per-track price of the iTunes Plus (DRM-free, 256kbps AAC) tracks are $.99, not the increased $1.29 price. See, for example, the new Iron & Wine album's listing. The album costs $9.99, but the individual tracks are still only $0.99.
• As of this writing, the FAQ has not yet been updated: it still says that "iTunes Plus songs are available at $1.29 per song."

Which came first, the agreement with the indies on the new price, or Amazon's MP3 store? From where I sit, this looks like a reaction to Amazon, which seems like a first for Apple with the iTunes store.

Oh, in case anyone is curious, but hasn't been following my Twitter feed, I got an iPod Touch last week. It's magnificent, and a delight to use. It's even more of a joy to demo it to my colleagues and friends around here. They're very impressed with the price (£199), as well.