Look, I essentially agree. I use hard drives for my own archival backup needs.

And if I were advising someone who was doing no backup whatsoever, I’d definitely advise them to pick up a hard drive and use (the awfully executed) Time Machine rather than doing nothing.

But that’s not the entire story.

For one, as Blu-Ray comes down in price, it’s going to become an attractive backup solution for everything other than video.

For two, the expense of doing hard drive backup CORRECTLY starts to get expensive.

In order to replicate the archival robustness I used to have with DVD backup using hard drives, I had to purchase a drive dock, Hudzees, and three hard drives. (If you’re not cycling backups between hard drives, you’re not doing it right. If you’re not keeping an additional backup off-site, you’re not doing it right.) An acceptable level of redundancy is MUCH cheaper to achieve with optical media.

In all, moving from DVD archival storage to hard drive archival storage done properly will cost the average Joe a minimum of about $400 to enter the game (and that’s assuming they are highly educated about how to best purchase their gear.) Not to mention that will be future costs as well.

“The only people I know cycling backups are Enterprise and that is a whole other topic entirely.”

If you’re not cycling your hard drive backups, you WILL lose data at some point. It’s precisely why hard drive backup is sorta sucky: you need to be paranoid about it and continually mange it, or it will end up biting you. Redundancy (including off-site redundancy) is cheap ‘n’ easy with optical media.

Of course, we may be assigning different priorities to long-term preservation of our data. I’m not officially “enterprise”, but I value my data far more than I value my hardware…

“While you may still see optical on computers in 10 years, I stand by my argument that it is dead for archival purposes.”

For the purposes of figuring out if you will be able to easily access your optically stored backups in 10 years, the continued presence of optical drives is the relevant issue. The popularity of optical backup is not particularly relevant.

I did all my backups in the 90′s using DVD-RAM not because I thought DVD-RAM would be a popular archival solution. I did it because I knew I’d likely be able to easily store my archived material and be able to easily access it in 2009. As an end user, that’s what you should be worrying about.

While you may still see optical on computers in 10 years, I stand by my argument that it is dead for archival purposes.

“Hard drives are actually kinda sucky for archival purposes, due to the need for regularly cycling through concurrent backups.”

I don’t see the need for the averaqe Joe to cycle concurrent backups. Software such as Time Machine and Retrospect take care of the details in the UI while all “Joe” must do is provide a reliable disk to store to which I have mentioned above. The only people I know cycling backups are Enterprise and that is a whole other topic entirely.

“And once solid state takes over, storage form factors and interfaces will change, so the hard drive you’re archiving things on today will no longer be connectable with standard equipment.”

True, form factors and interfaces will change in the future, but at minimum there is a good 3-5 year crossover period in which you will be able to move your backups to the new standard (as I did with IDE to SATA).

Again, I highly stress the point that just because optical (in 10, 20, or 30 years) is available does NOT mean that people will use it for archives. You simply overestimate the time and energy people are willing to devote to backups.

“you can just “burn and forget” with optical discs.”

I think the key word in this phrase is “forget”.

One other item worthy of note:

(Not to mention that the data on it will degrade rather quickly.)

When you take this point into consideration, you must acknowledge the inherent limitations of the average optical disc as well. How many people research disc quality before purchasing discs for archives, general storage, etc. There is a reason that Wal-Mart, Costco, etc do so well: Bang for the Buck. And most (if not all) consumers want the most product for their money. In almost every case, this means quantity, not quality. Therefore “Joe” goes out and buys the 50 discs for $20 since he is going to archive 10 discs every Sunday (yeah, right) for his meager 80 gig media library. Those discs he just purchased have a 2-4 year life span at best before oxidation or warping (because Joe also knows nothing about proper storage of optical either) his archives and he’s wondering why.

I am not saying that optical has no place. You are right, until we are all netbooted, there will be the need for discs. However, software delivery (aside from Operating Systems) will continue to migrate to the cloud negating the need for optical even further which will accelerate the introduction of new ways to install an OS on your system.

Hard Drives, to me, are still the clear winner for Joe. And while I have exercised some diligence on the subject for my own self preservation, I still consider myself a Joe. And I offer my experience to Joes on a daily basis as I try to get people to consider backups.

Although I can’t imagine asking them to devote the best part of a day to it……..

*sigh*

-systemsboy

Well, hard drives will disappear too. Solid state storage is where everything is going. And once solid state takes over, storage form factors and interfaces will change, so the hard drive you’re archiving things on today will no longer be connectable with standard equipment. (Not to mention that the data on it will degrade rather quickly.)

That’s where things will go in the long run, but of course, in the long run, we’re all dead. If you’re talking personal archival storage, you’re just worried about the medium-term. Let’s say 5 – 10 years.

And if you think the average computer won’t have an optical drive in 5 – 10 years that is backwards compatible with CD’s and DVD’s, you just don’t understand the commercial function of the optical drive.

It’s not just the fact that billions of long-lasting music CD’s and movie DVD’s will still be lying around in folks’ closets. It’s also that Blu-Ray is just taking off now. The lack of bandwidth available for last-mile delivery of HD video won’t get solved for years, and optical disc usage will persist for years after last-mile HD VOD is solved.

And that’s not even mentioning operating system software delivery. Until we’re all running our systems off of software netbooted from servers in Cupertino or Mountain View, computer makes are going to need to include optical drives to allow re-installation of system software. And again, even once netboot appears in part of the market , optical drives will also persist in the market for years afterwards.

In short, you’ll almost definitely have a backwards compatible optical disc reader 5 – 10 years from now. And the data put on DVD today will be less likely to degrade over the next ten years than the data put on a hard drive today.

Again, there are excellent reasons to go with a hard drive based backup system. But the idea that “optical drives are on their way out” is not one of them. If we enter a near-term future where Blu-Ray gets cheap, and if you aren’t archiving a video collection, then Blu-Ray will likely be a much better archival option than hard drives.

Hard drives are actually kinda sucky for archival purposes, due to the need for regularly cycling through concurrent backups. It requires continuous active management, while you can just “burn and forget” with optical discs.

Advances in hard drives have been very rapid recently. We’re seeing a proliferation of fast, cheap and readily available drives. We’re seeing advances in speed. But all the optical media camp has come up with is Blu-Ray, which requires a new player and all new discs. It has not caught on, and it’s not because it’s too expensive. It’s because it’s not good enough. There’s already something better, and we’re already using it in all our computers and thumb drives and cameras and iPods: hard drives.

As hard drives become the de facto storage medium for everything — audio, video and data — you will see optical discs disappear. VHS tape had a massive user base for well over a decade, but now you can barely find a VHS player to play your tapes, and it has completely disappeared from video outlets like Blockbuster within a few short years of the arrival of DVD. Such will be the fate of DVDs and CDs as well, as people start downloading their music and movies from the Internet (which, just to drive it home, runs on hard drives for storage). In fact, some form of hard drive is used for video storage in most of today’s video cameras. Not tape. Not optical. Hard drives.

Hard drives have snuck up from behind and won the format war. Now that they’re a commodity, there’s little reason not to use them for everything. And I think that’s what we’ll be doing in ten years.

But then, it remains to be seen.

I just want to say, too, that this was a great round of comments, one of the best. This is one of my favorite topics to think about and debate. I’m glad it stirs a lot of other folks’ passions as well. Thanks, everyone who wrote in.

-systemsboy

Just have it run every night (cron job?), and it can make an effortless daily backup with very little overhead for the history (so long as you’re not making massive changes on an ongoing basis).

It’s in the MacPorts tree or at http://rdiff-backup.nongnu.org/. You can either use it over the network (ssh) to a server with rdiff-backup also installed, or you can just do back up to a locally-mounted volume. There’s also a similar program called “duplicity” that’s completely client-side for backup to locations without the program installed (e.g. Amazon S3). Duplicity can also encrypt the reverse diffs before sending them out.

Hard drives by FAR using a dedicated hardware RAID. Redundant drives provide far more protection than discs simply because you WILL back it up not to mention drives do not get physically handled as discs require for data retrieval.

Even Blu Ray will never catch up to the storage needs of the average consumer who backs up their home movies, music and photo collection. Nevermind if they purchase or backup a movie collection.

By sheer convenience the hard drive continues to decimate optical for backups. I know of no one who religiously backs up to disc on a consistent basis. Take that same user and give them an automated backup program and few Terabytes worth of space and now you are talking.

Products are continually developed and marketed to help users achieve this (DROBO, ReadyNAS, etc.) and OS developers are working to make backups the last thing the user has to worry about (Time Machine comes to mind). While I do not use Time Machine myself, I do use a Highpoint RAID card and 4 1 TB drives that store all of my daily backups from each of my machines using Carbon Copy Cloner. I know I am not the average user in this respect, but neither is a guy who burns 10 DVDs every Sunday night. And I don’t sit in front of the computer to do it. It’s all automated. If a drive goes dead, I get an email as well as an audible alarm and I replace the drive. It then proceeds to go through an auto-rebuild sequence. So far, so good for 3 years. How many discs would I have burned by now if I backup everyday?

I have a similar setup for my brother using a Drobo for a year now. He is not computer savvy, but knows if the orange light comes on (and he gets the email as well), all he has to do is go down to Best Buy and get a new drive, pop it in and he is good.

Anyone using optical media can easily lose a weeks worth of data (because who backs up manually more than once a week?) and the more they backup, the more wasteful the process is from a Green standpoint. And I won’t even go into throughput advantages of HD vs. optical when it comes time for retrieval of data.

Optical is dead for backup. If it is not dead, it will be in 10 years. The constant influx of digital medium will force it out of people’s minds as their digital libraries continue to grow and their storage needs exponentially outpace the storage capacity of the optical drive.

R.I.P.

Optical media really does have some significant advantages over hard drives for truly archival storage. I have data originally recorded on DVD-RAM cartridges in the 90′s that I think will still be readable with installed equipment 10 years from now. That’s a useful trait.

And with the expected decrease in Blu-Ray media and equipment costs over the next year or two, Blu-Ray will become a very intriguing archival medium.

Unfortunately, that’s only really true if you’re not dealing with video.

Once video enters your archival needs, hard drives become the only practical solution. Blu-Ray, however, could be useful if you’re just backing up audio and one or two users’ computers.