@lindsay.at

[Update: last weekend, I had a chat with the local O₂ store manager. It turns out that some of my assumptions about unlimited data, and how they will be rolled out to customers, might be wrong. Some of the more spectacular claims made below should be taken in that light.]

I perform a lot of speculation on this blog. I think my record speaks for itself. With that pinch of salt out of the way, I dug into O₂'s pricing structure, and noticed a few things.

First off, let me state my bias: I am an Apple fanboy. I wanted to be excited about the O₂ launch of the iPhone. From an Apple-partisan point of view, I think that O₂ flubbed the launch. The carrier's intentions are clear, and somewhat understandable, but to someone who wants to see Apple technology in lots of people's hands, it was a disappointment.

The £269 price was as predicted. It sounds high when converted to USD, but it's in keeping with what the local market could conceivably bear, and is consistent with Apple's pricing here, and in relation to the rest of Europe. The British Pound is extremely strong, but there are no guarantees that it will stay that way. Apple cannot afford to be punished should the US economy unexpectedly recover.

No, the disappointment lies in the premium prices for the subscriptions. O₂ left no doubt that it sees the iPhone as an exclusive product meant to attract and capture high-end consumers from all other networks. They're not interested in disseminating the iPhone as widely as possible: they want to get high-tariff customers on the network, and to keep them for at least 18 months (which is now the standard "long" contract in the UK — the market is in transition from an earlier 12-month lock-in period for contracts).

Unfortunately for O₂, we can quantify the premium they believe they can place on the iPhone. O₂ is refreshingly open about their tariff policy, and has a "Simplicity" tariff that is a pay-monthly SIM-only tariff: if you don't want to upgrade your phone, you can put it on the Simplicity plan and be locked in at a much-reduced rate. If you want a non-Apple phone to go with your monthly tariff, you typically pay £10 extra, which we can take as pure phone subsidies.

In other words, if you buy a phone from O₂, you should expect a discount of approximately £180 from the hardware cost. O₂ knows how to profit from its network whether you do this or not.

Interestingly, the current Simplicity tariff corresponds exactly with the voice and text portions of the iPhone tariff:

minutes	200	600	1200
messages	200	500	500
SIM-only	£15	£25	£35
iPhone tariff	£35	£45	£55

The iPhone has been raising a lot of eyebrows because it's "unsubsidised": consumers are willing to pay a full retail price for the unique, full-featured phone, and flock to the exclusive network for the privilege. The premium that customers typically pay for a new phone is pure profit, shared by the carrier and Apple, in a mutually beneficial agreement. As we can see, the difference between O₂'s price for the iPhone tariff and a tariff unattached to a phone is £20. The only differentiators are the unlimited data and the unlimited Wi-Fi.

The unlimited Wi-Fi partnership with The Cloud was a smart move by O₂: The Cloud is well-respected and a market leader. Partnering with them automatically pushes iPhone users' most demanding data needs away from O₂'s anæmic EDGE network. A consumer could subscribe to unlimited Wi-Fi with The Cloud for £6.99/month. I could imagine that a corporate partner of The Cloud could easily cut a deal for wholesale prices of £5–£6 per bundled user, per month.

I think a tactical error from O₂ was pre-announcing unlimited data for all of their consumers. As the Guardian reported:

Will the unlimited data package be available for other users?

[O₂ CEO, Matthew] Key: We'll roll them out on October 1 to our other tariffs.

Boom. Unlimited data for the iPhone is taken out as a differentiator, and explicitly absorbed into O₂'s cost of operating their network. Ultimately, this is probably a good move on the part of O₂ in finding customers, but it doesn't help sell iPhones one bit.

[Aside: I believe the fair-usage policy of "1,400 internet pages per day" equates to 100MB per day, but the wording is spectacularly vague on that count.]

I was shocked and surprised at the rumours reported by the Guardian that surfaced on Monday that "O₂… will return to Apple as much as 40% of any revenues it makes from customers' use of the device." Put in the context of O₂'s tariffs, though, it starts to be plausible. Forty percent of £35 is £14.

I believe that the lion share – fourteen pounds per month – of the £20 iPhone premium (over O₂'s SIM-free fees) is dedicated to paying Apple. Nearly all of the remaining six pounds of the premium go toward paying for the Wi-Fi access partnership. O₂ might see a pound or so extra income per iPhone user, but it seems entirely plausible that they're making all their ongoing profit on the base network fees. O₂ will also earn some retail margin on the iPhones that they sell in their shops, which probably strikes the carrier as a tidy bonus.

[edit: I didn't make it clear: since the "iPhone premium" stays the same for all tariffs, I believe the profit sharing with Apple stays at a constant £14/month, no matter which of the £35, £45, or £55 tariff the customer chooses.]

So, in addition to whatever wholesale margins Apple gets on the iPhone, they get £252 in profit sharing per customer. That's plenty of reason to break the hearts of other potential carriers. O₂ clearly feels amply rewarded in making this deal, gets a lot of buzz, and captures desirable customers. Will consumers go for it? I think they will, but not to the same extent that they did in the US.

So, when last I blogged about the ZFS RAID server, I may have ended on a down note, suggesting disappointment. I hope readers will understand that's not the case.

When I started this project, I sat down and examined what was most important to me in the server.

My requirements:
• Saturate an aggegated 2× GigE link for sustained reads and writes
• Do it cheaply

My strong desires:
• ZFS for its reliability, redundancy, flexibility, and ease of use
• Maximise the amount of usable space

ZFS wasn't a requirement. It couldn't be: it's a solution, and defining requirements in terms of pre-ordained solutions is, at best, compromised. Maximising IOPS wasn't a first priority, sustained write performance was. Still, I want to have decent random seek performance, because there will always be a case where performance falls to that level.

I ran some additional Bonnie-64 tests to see the difference between the SATA controllers and their PCI-X buses. There was a small (a couple percent) but consistent difference between the two controllers. I believe one ran at 133MHz, and the other ran at 100MHz (but, to be honest, I don't know what tools I would use to verify such a thing). So I moved a disk controller from the 100MHz bus to a PCI-X slot on the shared 133MHz bus, and ran the same tests as before.

The results are as follows:

I perceive a strong levelling-off of streaming write performance, even lower than with the previous test. The peak for three 4+1 RAID-Z groups is 387.5 MB/s, while the peak for five 2+1 groups is 354.5 MB/sec. The mirrored scenario's limits are even lower, at 258.5 MB/sec.

The continuous read performance is even more interesting. Now, it's clear that the two controllers are maxing out a single, contended PCI-X bus where they hadn't before. The read limit is at 520 MB/sec. That, to me, sounds very much like one half of the throughput of a 64-bit, 133MHz bus (1064 MB/s). (It's within 2.5% of half that figure.) One conclusion could be that ZFS performs two reads for every block requested from disk, whether it be RAID-Z or mirror.

Taking a step back, should we find significance in the fact that one-third of our PCI-X bus throughput is 354.67 MB/s, while the most we could squeeze out of the 2+1 RAID-Z configuration was 354.5? It would certainly square with what commenter "mrb" stated: for 2+1 RAID-Z sets, expect 50% higher throughput on reads than writes.

The random seek performance doesn't yet tell me much other than the theory of IOPS scaling linearly with the number of vdevs or mirror disks simply does not hold on my system. Frankly, I'm stumped at how it only increases logarithmically. Well, at least it increases monotonically.

Let's pit theory against practice. I originally posted a crude, back-of-the-envelope model of read/write/random ZFS performance for 14 or 15 disks in 2+1, 4+1 or 2× mirror configurations. What happens to our experimental results when I factor out the (estimated) base performance of a single drive/vdev set?

		Sequential I/O
config	Random Reads	Read	Write	Capacity
RAIDZ: 3×(4+1)	3y	12z	12z	6.0TB
RAIDZ: 5×(2+1)	5y	10z	10z	5.0TB
mirror: 7×2	14y	14z	7z	3.5TB

	Random Reads ×90/sec	Sequential I/O ×72MB/sec
config		Read	Write	Capacity
RAIDZ: 3×(4+1)	2.3y	7.2z	5.3z	6.0TB
RAIDZ: 5×(2+1)	3.3y	8.8z	5.2z	5.0TB
mirror: 7×2	3.8y	10.2z	4.8z	3.5TB

It's clear that ZFS is demanding enough that it can hit the limits of the PCI-X bus on a poorly thought-out system. I can sketch out those limits on my own system, in some cases. It's also true that I could have chosen another motherboard with 2 independent 133MHz PCI-X buses, or gone with a PCIe solution that would have eliminated any concerns about bus bandwidth. In theory, with this many disks, I could be seeing twice the performance in some situations. However, I should look at the numbers: 390MB/s far exceeds my ability to get data into or out of the machine via the network.

The machine does what it is supposed to, and surprisingly affordably, too. Any "disappointment" I have is purely theoretical.

As a postscript, I should make a call out to anyone who would like further data with the 133+100MHz controller configuration. The server is leaving the workshop and going into the rack now, but the system will be under test for a few weeks more. Contact me via the comments, the contact form on this site, or the zfs-discuss list if you have a particular scenario you'd like me to run.

This is the last time I'll indulge in this sort of iTunes-HD speculation until at least January 2008. I will freely admit that it's colored by wishful thinking, but here are some thoughts about Tuesday's upcoming press conference at the Regent Street that allow me to hope that Apple could possibly advance the state of the art in Hi-Def video delivery.

I'm not denying an iPhone announcement. That would be a fool's bet. I just think there could be something else cooking as well.

1) It's been quiet on the new television season front. While there are a few US shows whose seasons have already started, the big series seem to be starting from Sunday (23 September). Last year, many pre-paid, discounted season passes were trumpeted and pushed on the iTunes store. The distinct impression I get is that the store is waiting for something.

2) As I've pointed out before, if Apple wants to include high-definition televisual content on its store, it is best done at the start of the US TV season. Changing mid-season will confuse and frustrate users. The next opportunity will be September 2008.

3) The BBC has a lot of high-definition content that's just begging to be more widely distributed than it currently is. "Planet Earth" is consistently cited as a best-selling title in both HD-DVD and Blu-Ray formats. Selling titles like "Hotel Babylon," "Robin Hood," and "Torchwood," for example, might actually benefit from the one-at-a-time taster format of the iTunes store (rather than the outlay required for a whole season on disc).

4) Partnering with content producers like the BBC may well be a way for Apple to soldier forward with its ambitious plans: Hollywood and the major American TV broadcast networks have been digging their heels in response to Apple's increasing power as a digital content distributor.

To recap, the major components of the ZFS storage server were:

• PCI Case's IPC-C3E-BAR65-XP-SAS 3U chassis w/16 hot-swap SAS/SATA2 bays
• 2× AMD Opteron 275 processors (dual-core, 2.2GHz)
• Tyan S3892 (K8HM) motherboard.
• 8GB ECC memory
• 2× 80GB boot drives (connected to the motherboard)
• 2× Supermicro AOC-SAT2-MV8 SATA 8 port RAID controller
• 16× 500 Gb Seagate enterprise class SATA Hard drives

This configuration is intended to act as a streaming multimedia recorder/server, with the most demanding disk I/O workflow intended to be writing a stream of data coming in from two GigE links. Imagine what it might take to store uncompressed high-definition video. There are likely to be other demanding tasks, but this was the most extreme.

Tyan's S3892 suffers from some ambiguous documentation. The PDF specification sheet states in the text that there are two 133MHz PCI-X slots, and one 100MHz slot, whereas the block diagram says they all run at 133MHz. The manual says nothing about it. There was no answer from emailing Tyan support. Using the block diagram as my guide, I split the two Supermicro controllers amongst the two PCI busses, and decided to start testing the hardware configuration to be sure.

Basically, I wanted to test and compare the model I previously blogged about.

For mirrored configurations:
• Small, random reads scale linearly with the number of disks; writes scale linearly with the number of mirror sets.
• Sequential read throughput scales linearly with the number of disks; write throughput scales linearly with the number of mirror sets.

For parity (RAID-Z, RAID-Z2) configurations:
• Small, random I/O reads and writes scale linearly with the number of RAID sets.
• Sequential read and write throughput scales linearly with the number of data (non-parity) disks.

Bonnie-64 was designed to turn up performance bottlenecks. That is precisely what I was looking for. Can I tell that one controller is on a bus that runs 75% the speed of the other? I could, in fact, but the overall combined performance was very decent. The tests did show limitations in my hardware, however.

I compared configurations from two to fifteen disks (I always want to have a hot spare running), with 2+1 RAIDZ vdevs, 4+1 RAIDZ vdevs, and (single) mirror vdevs. So each graph reflects fifteen runs of Bonnie-64:

• With the 2+1 RAID-Z: 3, 6, 9, 12, or 15 disks in the zpool,
• With the 4+1 RAID-Z: 5, 10, or 15 disks in a zpool, and
• With the mirror configuration: 2, 4, 6, 8, 10, 12, or 14 disks in the zpool.

All of the graphs measure the number of data disks (i.e., the total number of mirror disks, but only the number of non-parity disks for the RAID-Z configurations), or in the case of random seeks, the number of RAID/mirror sets. All of the tests were performed with 32GB test files: from the results, it's pretty clear we're exceeding any cache issues.

Block writes were always going to be the metric I was most sensitive to, because of the above-described workflow. You can see that there is a strong levelling-off of block write performance just below 390 MBytes/sec. The mirrored configurations increase their write speed at half the rate of the RAID-Z configurations, as we would expect from the slower writes indicated in the model. The "ideal" line is fairly arbitrary, as it's an extrapolation of performance from fairly few data points. It is, however, indicative of what the performance model might predict.

Sustained read performance is much less limited than with writes. The "ideal" line also has a 33% steeper slope than with the writes: it appears we consistently achieve four block reads in the same time as it take to do three block writes. Strangely, the 4+1 RAID-Z groups underperform by a fair bit (I can't comment as to the statistical significance, at the moment, but it seems fairly consistent). The 7×2 mirror configuration tops out at 735 MB/s on reads, which seems fairly decent.

I'll admit that the random seek performance figures baffle me a bit. Everything I've read so far suggested that random seek performance would scale linearly with the number of vdevs (or disks in the mirror). Instead, the numbers line up fairly well with a logarithmic graph. Am I running into lots of vibration? Am I hitting an unexpected bottleneck that's unrelated to data transfer over the bus?

This is a beast of a post already. I'll push this out to the world, and start writing up the next installment, wherein I note that one of the SATA controllers is, in fact, on a slower PCI-X bus, and what I do to fix it.

edit: All of this was on Solaris Express Community Edition, Nevada 70, with the ZFS boot patch applied.

I've had a lot more thoughts on Apple's recent moves since last posting. None is quite enough to post individually, but they seem to make for a decent set of thoughts.

---

There has been a wide range of prices thrown around as possibilities for a new pricing structure for television shows on iTunes. Prices from $0.99 to $4.99 have been quoted. I have only seen the prices reported at their face value and only compared to $1.99 for the existing standard definition purchases. Has no one else considered that $0.99 might be for a rental, and higher prices might be for High Definition?

Incidentally, people should not forget that HD television purchases have had an established price for nearly ten months: Xbox Live has offered 720p download purchases for $3.00 since last November.

---

Mike Lee's rant on the entitlement people felt on hearing about Apple's iPhone price drop was amazing, and expanded on my initial reaction perfectly. That said, Apple's premeditated response was perfect. I also see it as being a

1. one-time benefit to early adopters of any of Apple's products ("This is life in the technology lane." In other words, "I hope you've learned your lesson."), and
2. the product of a wide-ranged sociological experiment. (Would users take this sort of price drop? No? Okay, we've learned a lesson.)

Perhaps it was enabled by – or even the reason behind – Apple selecting a subscription financial model for the iPhone.

---

I am a bit stymied by the specificity of the naming of the "iTunes Wi-Fi music store." At first, I got it: Apple's not going to deal with delivery over the mobile telecom networks, and they're not dealing with video. Fine. It's verbose, but clear.

On second blush, however, I wasn't so sure. Why couldn't this be used for purchases over the AppleTV? The same underlying model (and controller) surely would serve the AppleTV's interface equally well. In fact, the seven major store headings ("New Releases," "What's Hot," "Genres," "Featured," "Top Tens," "Search," and "Downloads") line up fairly well with the YouTube interface headings ("Featured," "Most Viewed," "Most Recent," "Top Rated," "History," "Search," and "Log In").

I would have thought the ITunes Wi-Fi music store was a sure sign that similar functionality was coming to the living room. Perhaps it is, but under a different name: "iTunes AppleTV store." That gets around the music-or-video question as well.

---

Last week, I had thought the iPhone's closed API was because of the the mobile carriers. Apple didn't want to come up with a guaranteed API until it had all of its carrier agreements down, and there was no chance of the carriers pushing to have their own applications on the Springboard. Independent reverse-engineering and programming has been allowed because there's little chance of it appealing to institutions. Thanks to Ben Metcalfe, I now think that the iPhone interface may never be "open" in an official way:

Dave Winer has a good observation:

“Scoble wants an SDK so developers can create cool iPhone apps. Of course I do too. But I doubt it’s going to happen anytime soon. Look at all the deals they can do if they don’t. Starbucks wouldn’t need them if there was an SDK. And Tulley’s could do their own, as could Peet’s, and Whole Foods, etc etc. Apple wants all that business, I’m sure. And they want to be able to sell Starbucks an exclusive. They couldn’t if there was an SDK.”

When he announced the iPhone, Steve Jobs said there would be no SDK because you could do everything you needed to in a web-browser/web-development environment. Clearly that’s not the case - the Starbucks’ widget is not something that the rest of us can implement.

It’s bad enough that users will be forced to have Starbucks marketing on their iPhone/iPod Touch screen. It’s a kick in the face to have built that with hidden functionality that goes against the previous ideals that were made about openness of the platform.

Apple may well be trying to be the new "orifice" here, via its Applications and unique partnerships, to displace the mobile carriers' own entrenched positions.

Well, I'm glad I still own some Apple stock.

---

I also noticed the iPod TV-out picture suddenly got more complicated. The new iPod nano and the iPod classic will not output video to the television without a (new) cable or accessory including an Apple authentication chip.

Could this be the start of Apple's (public) implementation of a "secure signal path" for the purposes of complying with Digital Rights Management requirements of High-Definition video content providers?

Also recently: iPod classic vs. Flash, Where's the HD?, iPhone vs. iPod touch, and Technical feasibility on HD delivery.

I'm trying to figure out what Apple wants to accomplish with the iPod classic with a hard drive (the "lazy" iPod) versus the flash-based iPods (the "active" iPod). I'll make no bones about it: I have been waiting for a hybrid device with the capacity of an iPod classic and the wide screen of an iPod touch for over two years. I am frustrated that I have to choose between capacity and screen size. That said, I am (mostly) reconciled with the situation as it is today. I understand that the twin concerns of battery life and form factor make such a hybrid undesirable by Apple.

But why the move to flash? Doesn't it do Apple a disservice, where it gets people derailed from the track of massive capacity iPods? For people with large media collections, having a "lazy" iPod means that device management is largely deferred to the time you want to play your media. But once you decide you want the features from a flash-based device, you only need to intensively organize your playlists, smart playlists, and iTunes habits once, and (theoretically) you never have to deal with a high-capacity iPod again.

Maybe it's purely financial. The data density per (hard drive) spindle seems to be leveling out with the class of 1.8-inch drives. Maybe Apple saw this (pretty clearly, due to their massive investments in this area) and sees much more long-term growth and returns on moving people to flash memory. The iPods have progressed from 1GB to 16GB of flash much more quickly than from 10GB to 160GB of hard drive capacity.

The possibility that intrigues me most was first brought up by my officemate: what if Apple doesn't want lazy iPods (and lazy iPod users) out there? What if Apple wants to encourage people to dock their iPod more often? It could be very beneficial to re-engineer user behavior from synching only before major travel to synching regularly. Introducing podcasts may well have been an early form of this. Television season passes may also be seen as another way of accomplishing this.

Why would Apple push this behavior? Because each time a user syncs, the user interacts with iTunes, and each time that happens, there's a chance that the user interacts with the iTunes music store. It may well be that Apple's desire to get "active" iPod users has increased due to new services and/or security models. (e.g., DRM appropriate to movie rentals, perhaps?)

I live in hope.

Apple's announcement yesterday was about music. I had hoped that it would fill in the picture for high-definition video delivery to the home. I still think the pre-holiday season is the ideal time to deliver this, for Apple.

So, because it's my nature, I'm continuing to speculate on Apple's hi-def delivery possibilities until it does happen. I think if it happens this year, it'll have to happen soon, like, by 18 or 19 September by the latest. Why? The major US networks' television season starts in earnest on Sunday 23 September, and that will be a well-established, rich stream for ready high-definition content. The movie releases will probably be a trickle, but seeing downloadable television in hi-def will get consumers used to seeing it on their big-screen, flat TVs, and demanding it in greater quantities.

Since I'm engaged in reckless speculation, I may as well add that I believe that that sort of deadline is what led to Apple's particular urgency in the negotiations with NBC, and the public spat that followed. I think the $4.99 price point that has been the focus of the dispute, and has since baffled commentators, was the price that NBC was pulling for with HD downloads. Why, if NBC is doubling the wholesale price, would the end-consumer's price go up by 150%? Only if Apple's costs were increased. I'm guessing they would triple from where they are.

I imagine that Apple was pulling for a slight wholesale premium for high-definition television downloads. Maybe they were pulling for a $2.99 price, more likely they were willing to go up to $3.99. NBC may well have said, "If you're charging double, we want to charge you twice the price." The bandwidth costs, however, would eat into Apple's margins, and they reached an impasse, made particularly painful because Heroes in hi-def would be the ideal flagship launch title. Apple's response was to pull NBC's new season launches, to eliminate confusion about standard-definition downloads, living in hope that the NBC-hi-def picture would resolve before year- (or season-) end. (If a program is upgraded to HD mid-season, what happens to the existing downloads? A solution is technically possible, but it will cost money, and create a logistical nightmare for billing of – and communications with – customers.)

I want to believe that Apple had planned HD for yesterday's event (and had ABC, CBS, FOX, and the CW lined up), but pulled it in hopes that two weeks' more negotiation would make the difference with NBC. For once, I think that Apple does not hold all the cards at the negotiating table. They scored an early win in public opinion, but NBC's move to Amazon's Unbox service was a clever counter, confusing people with regards to Apple's stated prices. Ultimately, these public moves won't count for much during negotiations.

Me, I really am hoping to see Heroes in hi-def, offered on the iTunes store. I would pay a reasonable price for the season. So long as Apple negotiates the right price for the bundled package, I don't care so much what the individual episodes cost.

In the twitter commentary that followed Apple's iPod announcement, while I was getting caught up with the news, I realised that Mail was missing from the new iPod touch. I got to thinking that was an interesting omission, and reflected that the widgets, the stock, weather, and Google maps features, in particular, were particularly telling (rather than "curious"). It points to the fundamental (and very deep) way in which the iPod touch and the iPhone differ.

The iPhone is an always-on, lightly tethered communications device. It presumes that at least a minimal network connection is always available. Weather and stock information is conceivably continuously updated. Mail can be checked regularly or pushed to the device.

The iPod is about entertainment, as it always has been. It is the mobile "lean back" to the iPhone's "sit forward." The 'curious' thing about these two devices is driven by how the world is today: in 2007, an Entertainment device and a Communications device look remarkably similar. (And, as an interesting corollary, it exemplifies how Apple generally sees the web as it is today.)

And it's in that striking similarity and feature near-parity that the iPod touch necessitated a price drop from the iPhone, to something more in line with its feature differential. Really, you're paying subscription fees to AT&T wireless for having the communications device cum phone.

It sounds like a reasonable choice for people to make during the upcoming holiday season.