Bytes: I Am Jack's Catastrophic Hard Drive Failure.

On Saturday morning my hard drive stopped.  It is less than a year old, but moving parts are moving parts, and its parts had problems.  I thought for the sake of other small business owners, especially those running OSX, I would catalog exactly what my previous backup solution was, what I lost, what was saved, and my new backup and recovery arrangement.  Partly this is for my own reference, but it might be helpful for others as well!

The Old System

We run a couple different businesses out of our home.  One is a teensy little game studio, where I work on iPhone and Flash content and engineering.  The other is a teensy photography studio, where my wife does family portraits and weddings.  Our primary work computers are MacBooks, so Time Machine was an obvious choice for backups.  For extra security, I made sure we were both using an external hard drive for our backups.  Since we both work out of the same room of the house, I just got a USB external, and whenever OSX gripes about us not doing backups, we just plug in the device and away it goes.  Easy, cheap (less than $200 over a year ago), and relatively effective.  Kind of annoying though, as you have to remember to eject the drive when the backup is completed, or risk trashing the backups by pulling the cord without unmounting the hardware first.

What Was Lost

My last backup was about 10 days ago (Time Machine pesters you on the 11th day, sadly).  I lost my VMWare Windows XP partition, one iTunes album, a couple of photoshop files, and some homebrew NES ROM stuff.  All my work files are backed up via git to a server in a basement in Vegas, so I was safe there.  Also, obviously, I lost the hard drive, which required careful use of a dremel, $100, an hour of driving, and 20 minutes of vile cursing to successfully replace.

The Restoration

Once the new hard drive was installed and the basic OSX install was completed, I simply checked "transfer settings from Time Machine" and hit continue.  A few hours later, my laptop was almost completely back to normal.  All my FTP, Bookmarks, cookies, instant messaging, documents, and music were exactly where I'd left them.  It saved my printer queue, my applications, just about everything.  Except...

I had to spend about one hour downloading and installing system updates, and creating the first new backup on the Time Machine is taking a very, very long time.  I had to re-authorize iTunes, and I have no way to de-authorize the old system, so one of my five authorizations may be gone forever.  While all my Windows documents were stored in OSX, I do have to completely reinstall my XP virtual machine and the apps that I use in that system. Time Machine does not handle the giant 40GB block of virtual machine that VMWare generates very well; if you try to include it for backups, you can pretty much obliterate your Time Machine in no time at all.  Finally, XCode is messed up, though I have no idea how or what is wrong with it.  It is a literal question mark in my dock.  Reinstalling that will suck, it is at least a 2GB download.  Update: verified that XCode is well and completely GONE from my system.  No idea why!  2GB download ensuing...

Lessons Learned

Ideally, especially for a business, losing a critical piece of hardware should be something you can repair overnight, with NO loss of data.  Some losses are unavoidable (the virtual machines especially), but can be repaired in an hour or so with no great pain.  So if you run a small business, with distributed, wireless computers (especially laptops), and you don't have buckets of corporate overhead budget to blow, how do you keep everybody's systems safely backed up?

Apple would prefer you invest in one of their Time Capsules, a $300 device that is both a NAS (network-attached storage) and a wireless router.  With a terabyte of storage, as long as you're not hording pornography you can keep months of backups for at least 4 computers.  Since it supports wireless, your laptops or computers can be connected and backing up from anywhere in the house/building/etc.  Unfortunately, this device is susceptible to the same types of physical problems that your laptop or desktop hard drive might run into.  That is, one day the Time Capsule could seize up and you will permanently lose those backups forever.

Reluctantly I decided to start doing some research into some alternatives.  The first thing I discovered is that Time Machine is basically incompatible with most NAS devices.  You can run some scripts that make those devices visible, but when you want to restore your data you have to manually mount ummm sparse bundles or something from the terminal.  The main issue is that Time Machine is designed to work best with AFP, Apple's file sharing protocol, and most NAS run SMB, for windows and linux compatibility.  Regardless, it begins to defeat the whole purpose of the fast recovery operation.

Eventually I was able to put together a more secure terabyte solution for about $400, just a little more than Apple's device, and quite a bit less than Apple's two terabyte solution.  My requirements for the solution are that it be accessible over wireless, and secure enough that if there is a physical problem with the device, I do not lose all my data.  I found out what all sysadmins already know, which is I was looking for a RAID1 configuration.  This is a special way of setting up a pair of hard drives so that all the data from one drive is "mirrored", or duplicated, on the other drive.  Any physical failure you might have in one of the drives would be relatively inconsequential, as all the data would still be fully accessible from the other hard drive.  Time Capsules, of course, are not mirrored!

The New Hotness

I was still stuck on the whole "incompatible with most NAS" problem, though, until I found out that the new Apple Airport Extreme had integrated support for Time Machine backups through its external USB port.  That meant all I had to do was find a cheap RAID1 USB backup, copy the old Time Machine to the new one, and we would have a really solid solution.  After blowing $180 on the Airport Extreme, I found this well-received Western Digital RAID1 external drive for $230.  So it ended up costing us a little more than Apple's own Time Capsules, but the result is just as easy to use, vastly more secure, user serviceable, and only runs you an extra $100.  I also spent $50 on a simple UPS, to prevent air-conditioning-triggered brownouts from killing the Time Machine in mid-backup, potentially corrupting the whole drive.

For under $500, we have a wireless, mirrored backup and recovery system that is not even susceptible to power outages, and can completely restore a repaired or replaced Mac computer in a matter of hours.

Update: I found this advice helpful when plugging a Time Machine into the Airport Extreme, since OSX wasn't finding it very easily on its own:

1. Connect your USB disk directly to the Mac.
2. Set up Time Machine and let your initial backup run.
3. Eject the disk and connect to the AEBS.
4. You need to ‘touch’ the disk by opening it in Finder, then Time Machine realises where it is and carries on.