@lindsay.at

I'm trying to set down the steps that I did to enable me to do Time Machine backups to an AFP store on top of ZFS running on OpenSolaris. It just happens to be the combination I'm running. There's no real need to combine all these steps as they apply to you: I just want to make note of what I had to do.

First, I installed netatalk, mostly working from the instructions listed at confessions of a unix junkie. Since Leopard strongly deprecates sending AFP passwords in the clear, I had to build with OpenSSL. Since I used pkgsrc to build the dependencies, my ./configure command looked more like:
LDFLAGS=-R/opt/local/lib RANLIB=echo CC=gcc ./configure --prefix=/opt/local --without-pam --disable-ddp --disable-tcp-wrappers --disable-srvloc --with-bdb=/opt/local/include/db4 --with-cnid-dbd-backend --with-ssl-dir=/opt/local

The Diffie-Hellman Exchange UAM (uams_dhx.so) is fairly critical to use with Leopard, so the "unix junkie" recommendations for the cleartext password should be ignored.

With that accomplished, set up the backing store. For me, it's a ZFS dataset (filesystem) on a centralised mount point. Make sure that the owner points to the user who will be using the AFP share (with a chown). Make sure the AppleVolumes.default entry for each share points to the right directory and user, such as:
/Storage/atl-backup "TimeMachine" rwlist:atl allow:atl

Once the mac could connect securely, there were only two steps needed. Allow Time Machine to deal with unsupported AFP shares:defaults write com.apple.systempreferences TMShowUnsupportedNetworkVolumes 1and tickle the mounted share into being recognised for this unsupported hack:
touch /Volumes/TimeMachine/.com.apple.timemachine.supported

That was enough to get the backup started. As I'm still running the initial backup, I have no idea of the stability of this solution, but I hope to be able to report back. I also notice that reports say you must mount the "sparsebundle" disk image manually in order for the Time Machine recovery GUI to be of use.

I've written before about what a ZFS-based storage appliance might look like. I think I've glimpsed a first step in that direction. Yesterday, by chance, I came across Norco's recently-announced DS-520, a build-your-own barebones storage server. You provide the operating system, disks, and RAM. I hadn't seen such an open, x86-based platform in that form factor before, so naturally my mind wandered towards OpenSolaris.

My first concern was the chipset compatibility. Norco emailed me back quite promptly saying that the Marvell 88sx6081 was the SATA chipset they used. Cool. That's what goes into the Thumper, so Solaris support seems trivial, at least for the one component that might cause the most trouble.

It sounds like the makings of a very nice home-server platform:

• $729 for the 1GHz / 2×GigE unit
• add 1 GB RAM,
• a fast CF card for the OS, and
• five 750GB disks (looks like they're on the right side of the price/capacity "knee" now), topped off with
• OpenSolaris stripped down to a fairly minimal set

Not cheap, or trivially easy at this point, but it could make for an interesting project, and a great piece of kit to sit on the shelf.

I'm trying to figure out what Apple wants to accomplish with the iPod classic with a hard drive (the "lazy" iPod) versus the flash-based iPods (the "active" iPod). I'll make no bones about it: I have been waiting for a hybrid device with the capacity of an iPod classic and the wide screen of an iPod touch for over two years. I am frustrated that I have to choose between capacity and screen size. That said, I am (mostly) reconciled with the situation as it is today. I understand that the twin concerns of battery life and form factor make such a hybrid undesirable by Apple.

But why the move to flash? Doesn't it do Apple a disservice, where it gets people derailed from the track of massive capacity iPods? For people with large media collections, having a "lazy" iPod means that device management is largely deferred to the time you want to play your media. But once you decide you want the features from a flash-based device, you only need to intensively organize your playlists, smart playlists, and iTunes habits once, and (theoretically) you never have to deal with a high-capacity iPod again.

Maybe it's purely financial. The data density per (hard drive) spindle seems to be leveling out with the class of 1.8-inch drives. Maybe Apple saw this (pretty clearly, due to their massive investments in this area) and sees much more long-term growth and returns on moving people to flash memory. The iPods have progressed from 1GB to 16GB of flash much more quickly than from 10GB to 160GB of hard drive capacity.

The possibility that intrigues me most was first brought up by my officemate: what if Apple doesn't want lazy iPods (and lazy iPod users) out there? What if Apple wants to encourage people to dock their iPod more often? It could be very beneficial to re-engineer user behavior from synching only before major travel to synching regularly. Introducing podcasts may well have been an early form of this. Television season passes may also be seen as another way of accomplishing this.

Why would Apple push this behavior? Because each time a user syncs, the user interacts with iTunes, and each time that happens, there's a chance that the user interacts with the iTunes music store. It may well be that Apple's desire to get "active" iPod users has increased due to new services and/or security models. (e.g., DRM appropriate to movie rentals, perhaps?)

In the course of hanging out on the ZFS discussion list, I've developed a healthy paranoia for the safety of my data. Hard disk drives fail all too easily nowadays, and I know it's a matter of time before another major data-loss event, especially since my family's large media collections are being held on single external hard drives prized more for their attractive cases than for their reliability.

I found out a had a few hundred pounds left on my consulting account code at the university, and it's ideal for buying computer kit. I ran it past my Executive Producer (read: 'wife'), and she's also ready for redundant network-attached storage, having also looked into it for her job. I spent several days trying to figure out if I could build a solution myself based on OpenSolaris and ZFS, but ended up thwarted by form factor: I could find no cases that were compact as well as capacious enough for (at least) four hard drives.

So, I looked around, and it sounded like the Infrant ReadyNAS NV Plus is the unit to beat. As a UK supplier has a special on disks bundled with the unit, I put in an order right away.

But I wonder, what do I really want? I want flexibility and ease of administration. I want verifiable redundancy. I want ZFS. I also want something that will work seamlessly with my home network. I want something that understands Apple's protocols well. I want to be able to stop running iTunes at some point during the day, but still keep synching with my AppleTV. I want an integrated backup solution that works well with Apple's upcoming Time Machine feature in Leopard. I want something only Apple could provide.

The net is strewn with "Apple needs to provide this form factor or they won't get my custom" articles. I know that Apple's strength is in providing a simple, comprehensible product range that potentially serves a vast proportion of the populace. Still I wonder if an "iServe" could be coming. Consider:

• The AppleTV, iPhone, and the Airport Extreme (presumably running Darwin) are showing the way that Apple can expand their offerings while cashing in on the intellectual capital they've built up over the years of fine-tuning the full stack of Mac OS X. Add a little more sophistication with regards to storage to the Airport Extreme, and you have the basis of the device I'm envisaging.
• Apple has been building expertise and goodwill in the storage arena for years with the XServe RAID. It gets a fair amount of respect for that product from certain quarters. It won't be the first time Apple has tested a technology at the professional high end as a dress rehearsal for something in the consumer space.
• Apple has been very open about its efforts to port ZFS to Mac OS X/Darwin, and many have noted how ideal Time Machine would be as a GUI for ZFS's snapshot feature.
• The backup qualities of Time Machine only go as far as the reliability of the backup media, and external hard drives have had a sinking reputation for years. Building a reliable RAID NAS on ZFS would be a solution to a problem that people don't realise they have. You can now have faith in your backups, and serve all your home's or small workgroup's computers at once.
• Time Machine, as far as I understand it, seems to be rather storage-hungry (with snapshots, saving a copy of every version of every file that has existed on your computer from the day you install Leopard), and with the vast majority of Macs shipping with a single hard drive, that's a bit incongruous. Why should Apple sell you a software solution without providing the hardware or service to go along with it? This could be a sign for Apple to sell you something more.
• With iTunes moving into an increasing role as a media hub application, it makes bigger and bigger demands of the machine it runs on. With an AppleTV in the house, the ideal (as I use it) really seems to be 24-7 operation of the host computer, with iTunes always running. My Mac mini G4 has always been a 24-hour computer, but other people would like to send their iMacs to sleep once in a while. Seeing as iTunes serves multiple roles (media jukebox, media organizer, store front end, and a synchronizing/streaming server) at once, there's room for splitting off one of these functionalities into a lightweight server process that integrates with the iPod ecosystem more cleanly than the third-party daapd offered with RAID NAS products.
• There are NAS and RAIDed NAS products on the market already, and although they're gaining popularity, there are no dominant players. That's an opportunity for Apple to capture mind share with a new product. It's interesting that Netgear just bought Infrant, but Netgear still doesn't seem to be able to capture the public's imagination as Apple does.

RoughlyDrafted had a nice analysis of Apple's current NAS storage capabilities when it compared the new Airport Extreme with Windows Home Server. I think the Airport-as-NAS is just the thin end of the wedge for Apple, so here's what I propose:

• A 4- or 5-disk RAIDZ-capable array of disks
• AFP and SMB file sharing
• Printer sharing
• The server portions of iTunes
• Low power consumption
• Small form factor
• Simple administration through something resembling the Airport Assistant
• Possibly a gigabit ethernet interface (but certainly at least 100Mbit), for workgroups or direct connections
• Possibly USB and Firewire for direct connections

...and that's it. It could be a wireless (802.11 draft N) device, and could have router and wireless access point capabilities, but those would dilute the message on what this device is for: to serve and protect your files, reliably, quietly, and cheaply.

Everyone wonders when, with Apple. This is mere speculation: it may never happen. When could it happen, then? I would expect this sort of product to come in the wake of next year's Macworld Expo in January 2008, at the very earliest. I see the iServe as dependent on Leopard shipping, to help build the demand for the feature through Time Machine. It wants a major announcement to give it the time in the spotlight for the general populace to understand what it's for. That suggests Macworld Expo. As I've envisaged it, it's quite a boring, back-room product (which, we have to note, resembles the new Airport Extreme in 2007, which didn't even get a Macworld keynote mention), so an October release for the Christmas buying season would be inappropriate. I know that I could use this sort of product tomorrow, and would happily cancel my Infrant order to do so. That, I must acknowledge, is more wishful thinking than anything else in this article.

Any thoughts?

[I know I'm not the first to propose an iServe (though these couple links were found just after I finished the article). This is ultimately a revisiting of an idea that's been around for a while, with an idea on how Apple could make its mark on an emerging market category, and a survey of current developments with the Apple of 2007.]

I've been spending a fair bit of the last week puzzling through the various postings on ZFS performance. While Richard Elling's blog posts were informative, they didn't really tell me much about the workflow that interested me most: high-throughput multimedia streaming. I eventually took the question to the ZFS-discuss list, and got a lot of knowledgeable feedback. The essence of what I learned about various ZFS configurations, for my purposes (once I understood the superficial size/reliability/performance tradeoffs), gets boiled down to one choice:

Which is more important to your ZFS workflow:
random access or write performance?

I suppose this is old hat to people who are very familiar with RAID systems and/or ZFS, but it took some digging for me to find out. I'll set it out in words, in contrast with the pretty but dizzying graphs at relling's site. These guidelines obviously set aside any other bottlenecks, but as was consistently pointed out, media speed is usually the bounding factor with performance.

For mirrored configurations:

• Small, random reads scale linearly with the number of disks; writes scale linearly with the number of mirror sets.
• Sequential read throughput scales linearly with the number of disks; write throughput scales linearly with the number of mirror sets.

For parity (RAID-Z, RAID-Z2) configurations:

• Small, random I/O reads and writes scale linearly with the number of RAID sets.
• Sequential read and write throughput scales linearly with the number of data (non-parity) disks.

In other words, mirrors suffer on writes, collapsing to the number of mirrors, essentially. RAID-Z groups suffer most with random I/O, collapsing to the number of RAID groups, performance-wise, in those situations. A hypothetical table with two different configurations of 12 disks (four 3-way mirror sets vs. two RAID-Z2 sets) helps show the strong contrast:

	Random I/O		Sequential I/O
config	Read	Write	Read	Write
mirror: 4*3	12y	4y	12z	4z
RAIDZ2: 2*(4+2)	2y	2y	8z	8z

where y is the number of random, short IOPS and z is the sustained media throughput on the drives

My case is fairly clear: I want to both read and write multimedia streams fairly equally, so I favour RAID-Z groups. I don't need the same sort of long data life that others do, so I set RAID-Z2 aside for now.

In my particular case, I have 16 500GB SATAII drives to work with for the RAID. I am committed to one hot spare, so I'm down to 15 drives. Once I get my server, I need to know how much—and when—performance degrades when excessive numbers of streams are added and/or Random I/O requests are added to the mix.

For a long time, I had assumed I would use three sets of five-drive RAID sets. Looking at three-drive sets, I have to consider whether a ~17% drop in peak streaming performance is worth a 67% improvement in baseline small, random I/O (essentially the worst-case scenario).

How do I arrive at that? Five 2+1 RAIDZ groups have 10 data disks compared to the 12 in three 4+1 RAIDZ groups. If I go from 4+1 to 2+1 groups, I lose 2 disks worth of data storage and the equivalent amount of max streaming capacity, but gain two more RAIDZ groups for working on the seeking for random I/O. Actually, another table really makes the picture quite clear. I throw in a set of mirrored drives as further food for thought.

	Random I/O		Sequential I/O
config	Read	Write	Read	Write	Capacity
RAIDZ: 3*(4+1)	3y	3y	12z	12z	6.0TB
RAIDZ: 5*(2+1)	5y	5y	10z	10z	5.0TB
mirror: 7*2	14y	7y	14z	7z	3.5TB

TextDrive has introduced a new product, Bingo!. 100 GB of solid, redundant, networked storage for $199 per year. It's an interesting service model. It's about three times the cost of Amazon's S3 storage, but much more usable, and most likely more solid/redundant. Interesting thing.

I love using their StrongSpace product, but I think that's largely a product of having paid for a "lifetime" account package, which was essentially 20GB of secure storage for life, for $200. (Mixed Grill upgrade, for those keeping score at home. I see the Joyent as an interesting bundle thrown in "for free," which is exactly what they're doing with the Bingo! plans, as well...) Paying that amount every year for five times the space? I think I need to be convinced of that.

Rental versus ownership, I think, is the issue here. I'll need to comment more later.

(See also the Joyeur Weblog post on it, as well... and TextDrivers can pick it up at a discount using a code from the forums.)