Let’s face it, my data is everywhere. I have drives all over the place for various and sundry purposes. Some are full. Some aren’t. I also have a folder for things I plan on trashing eventually, when I need to reclaim some space, but want to keep around just in case, called TrashMe. And I have a folder with items I plan on backing up some day called BurnMe. This name comes from the fact that traditionally I archived up my data to optical media: either DVDs, or, in the distant past, CDs, so I also have several containers full of optical media.
This isn’t working for me anymore.
Optical media is no fun to use for backups. It’s slow, requires keeping a stock of media on hand, and it’s often undersized for today’s hefty data needs. For instance, backing up my current BurnMe folder will require about fourteen DVD-R discs. Burning and verifying those DVDs will take as many if not more hours. And then it will all need to be cataloged somehow, which is also a lengthy process. Finally, retrieval is almost as slow and tedious: find the disc you need in the catalog, get it from storage, load the disc up and copy the data back to the computer, which can also take quite some time. And if, heaven help you, there’s one tiny scratch on that disc you could lose all that data.
So I’m switching to a better way: hard drives.
One of the best bangs for your buck per-gigabyte of storage is, of course, the hard drive. In addition to being cost-effective, they’re fast, they hold a lot of data, they’re read-write and extremely versatile. You can use them internally, depending on the drive and computer in question, of course. Or you can use them externally in any number of ways, the most obvious being in a firewire or USB case. Using such a case allows for the attachment of the drive to just about any computer you can get your hands on. Drives don’t require much physical space, and because they’re so fast, they’re quick and easy to catalog and restore from. In fact, using hard drives as an archive solution entails little difference from accessing local storage. The hardest part is getting the drive off the shelf and getting it attached to your machine. But there are hardware solutions to simplify that process as well. Hard drives, which have largely remained the same since I began working in this business ten years ago, are more future-proof than optical media, which is constantly the subject of format wars and which is often subject to a raft of compatibility issues. Put another way: getting data off a hard drive in the next ten years is more likely to be supported by my current hardware than getting it from a DVD.
So, my plan, going forward, is to use hard drives to archive all non-essential data. I will continue to archive certain critical files to DVD in addition to the hard drive archive. But most of my stuff will be on hard drives. Things like bittorrent backups and movie files, old audio, video and web projects, images and what have you will stay on hard drives earmarked for the archive. Once one of those drives is full and I’m no longer using the data, I’ll burn anything critical — finished projects and their assets, for instance — to optical media, catalog the drive with CD Finder or similar software, and then put the drive in a Hudzee and up on the shelf. Done and done!
This greatly speeds and simplifies my archive procedure. And, because of the increased space and time efficiency, I can archive a lot of data I would have thrown out in the past. I still have yet to work out the details of this system, but I think it will be a vast improvement over the old burn-and-catalog weekends of my not-so-distant past.
If anyone has thoughts on how best to archive data I’d love to hear them in the comments.
10 Comments
“Put another way: getting data off a hard drive in the next ten years is more likely to be supported by my current hardware than getting it from a DVD.”
This is nuts.
Don’t get me wrong. My archival procedure is hard drive based due to cost and convenience reasons. But the likelihood of being able to access DVD’s with current hardware in 2019 is very high. Optical drives will almost definitely still be in fashion in 2019, and they will almost definitely be backwards compatible with current DVD’s. The economics of the installed base will make it so. It’s no coincidence that a CD burnt in 1999 is still readable with current hardware in 2009.
And that’s not touching on two other items: 1) a HD from 1999 is not accessible today without a specialized converter dock. 2) a HD from 1999 is far more likely to have experienced data loss from media degradation than a CD burnt in 1999.
Again, I agree that the price and convenience make hard drives the best backup solution, but that’s only because it’s cheap and easy to have concurrent backups. In other words, hard drive archival demands cycling your backups over time to new hard drives with fresh magnetic media and evolving HD interfaces.
But that doesn’t change the fact that a DVD burnt today is more likely to be readable with current hardware in 2019 than a SATA II hard drive.
—–
“If anyone has thoughts on how best to archive data I’d love to hear them in the comments.”
You’re on the right track.
- Drive dock (VoyagerQ is nice)
- Hudzees
- 1.5TB bare drives
- Redundant backups
- Redundant off-site storage
I use redundant drives for everything. One time a month I clone and keep in sync the hard drives using superduper. I also perform an hash check, using md5deep, for preventing data degradation.
One thing I’ll expand upon slightly:
Optical media really does have some significant advantages over hard drives for truly archival storage. I have data originally recorded on DVD-RAM cartridges in the 90′s that I think will still be readable with installed equipment 10 years from now. That’s a useful trait.
And with the expected decrease in Blu-Ray media and equipment costs over the next year or two, Blu-Ray will become a very intriguing archival medium.
Unfortunately, that’s only really true if you’re not dealing with video.
Once video enters your archival needs, hard drives become the only practical solution. Blu-Ray, however, could be useful if you’re just backing up audio and one or two users’ computers.
My 2 cents.
Hard drives by FAR using a dedicated hardware RAID. Redundant drives provide far more protection than discs simply because you WILL back it up not to mention drives do not get physically handled as discs require for data retrieval.
Even Blu Ray will never catch up to the storage needs of the average consumer who backs up their home movies, music and photo collection. Nevermind if they purchase or backup a movie collection.
By sheer convenience the hard drive continues to decimate optical for backups. I know of no one who religiously backs up to disc on a consistent basis. Take that same user and give them an automated backup program and few Terabytes worth of space and now you are talking.
Products are continually developed and marketed to help users achieve this (DROBO, ReadyNAS, etc.) and OS developers are working to make backups the last thing the user has to worry about (Time Machine comes to mind). While I do not use Time Machine myself, I do use a Highpoint RAID card and 4 1 TB drives that store all of my daily backups from each of my machines using Carbon Copy Cloner. I know I am not the average user in this respect, but neither is a guy who burns 10 DVDs every Sunday night. And I don’t sit in front of the computer to do it. It’s all automated. If a drive goes dead, I get an email as well as an audible alarm and I replace the drive. It then proceeds to go through an auto-rebuild sequence. So far, so good for 3 years. How many discs would I have burned by now if I backup everyday?
I have a similar setup for my brother using a Drobo for a year now. He is not computer savvy, but knows if the orange light comes on (and he gets the email as well), all he has to do is go down to Best Buy and get a new drive, pop it in and he is good.
Anyone using optical media can easily lose a weeks worth of data (because who backs up manually more than once a week?) and the more they backup, the more wasteful the process is from a Green standpoint. And I won’t even go into throughput advantages of HD vs. optical when it comes time for retrieval of data.
Optical is dead for backup. If it is not dead, it will be in 10 years. The constant influx of digital medium will force it out of people’s minds as their digital libraries continue to grow and their storage needs exponentially outpace the storage capacity of the optical drive.
R.I.P.
I’m not sure if I’ve brought it up here before, but I use rdiff-backup at work and it’s great. It’s a python program that uses rsync libraries to store 1) a copy of the mot recent backup as a regular rsync mirror, as well as 2) all of the reverse-diffs of files in previous backups so you can “rewind” a file or folder to whatever backup state you wish.
Just have it run every night (cron job?), and it can make an effortless daily backup with very little overhead for the history (so long as you’re not making massive changes on an ongoing basis).
It’s in the MacPorts tree or at http://rdiff-backup.nongnu.org/. You can either use it over the network (ssh) to a server with rdiff-backup also installed, or you can just do back up to a locally-mounted volume. There’s also a similar program called “duplicity” that’s completely client-side for backup to locations without the program installed (e.g. Amazon S3). Duplicity can also encrypt the reverse diffs before sending them out.
I’ll weigh in here, finally (been out of town): I’m pretty convinced that optical is dead. It will fairly quickly be replaced by hard drives or hard drive-like technology such as SD cards and SSDs. As those technologies become cheaper and more ubiquitous, DVDs and, yes, even CDs will fade away. The installed CD and DVD user base is significant, but it consists mainly of people who’ve purchased albums and movies, respectively, not of people who burn data to these media for archive. And people who have bought audio CDs and video DVDs are fast learning that there is a better way: hard drives. When was the last time you listened to — or even mounted for that matter — a CD? Hard drives offer advantages over optical in almost every way, except, perhaps, reliability (though hard drives are probably more fault-tolerant — 1 scratch can ruin an entire optical disc).
Advances in hard drives have been very rapid recently. We’re seeing a proliferation of fast, cheap and readily available drives. We’re seeing advances in speed. But all the optical media camp has come up with is Blu-Ray, which requires a new player and all new discs. It has not caught on, and it’s not because it’s too expensive. It’s because it’s not good enough. There’s already something better, and we’re already using it in all our computers and thumb drives and cameras and iPods: hard drives.
As hard drives become the de facto storage medium for everything — audio, video and data — you will see optical discs disappear. VHS tape had a massive user base for well over a decade, but now you can barely find a VHS player to play your tapes, and it has completely disappeared from video outlets like Blockbuster within a few short years of the arrival of DVD. Such will be the fate of DVDs and CDs as well, as people start downloading their music and movies from the Internet (which, just to drive it home, runs on hard drives for storage). In fact, some form of hard drive is used for video storage in most of today’s video cameras. Not tape. Not optical. Hard drives.
Hard drives have snuck up from behind and won the format war. Now that they’re a commodity, there’s little reason not to use them for everything. And I think that’s what we’ll be doing in ten years.
But then, it remains to be seen.
I just want to say, too, that this was a great round of comments, one of the best. This is one of my favorite topics to think about and debate. I’m glad it stirs a lot of other folks’ passions as well. Thanks, everyone who wrote in.
-systemsboy
“As hard drives become the de facto storage medium for everything — audio, video and data — you will see optical discs disappear.”
Well, hard drives will disappear too. Solid state storage is where everything is going. And once solid state takes over, storage form factors and interfaces will change, so the hard drive you’re archiving things on today will no longer be connectable with standard equipment. (Not to mention that the data on it will degrade rather quickly.)
That’s where things will go in the long run, but of course, in the long run, we’re all dead. If you’re talking personal archival storage, you’re just worried about the medium-term. Let’s say 5 – 10 years.
And if you think the average computer won’t have an optical drive in 5 – 10 years that is backwards compatible with CD’s and DVD’s, you just don’t understand the commercial function of the optical drive.
It’s not just the fact that billions of long-lasting music CD’s and movie DVD’s will still be lying around in folks’ closets. It’s also that Blu-Ray is just taking off now. The lack of bandwidth available for last-mile delivery of HD video won’t get solved for years, and optical disc usage will persist for years after last-mile HD VOD is solved.
And that’s not even mentioning operating system software delivery. Until we’re all running our systems off of software netbooted from servers in Cupertino or Mountain View, computer makes are going to need to include optical drives to allow re-installation of system software. And again, even once netboot appears in part of the market , optical drives will also persist in the market for years afterwards.
In short, you’ll almost definitely have a backwards compatible optical disc reader 5 – 10 years from now. And the data put on DVD today will be less likely to degrade over the next ten years than the data put on a hard drive today.
Again, there are excellent reasons to go with a hard drive based backup system. But the idea that “optical drives are on their way out” is not one of them. If we enter a near-term future where Blu-Ray gets cheap, and if you aren’t archiving a video collection, then Blu-Ray will likely be a much better archival option than hard drives.
Hard drives are actually kinda sucky for archival purposes, due to the need for regularly cycling through concurrent backups. It requires continuous active management, while you can just “burn and forget” with optical discs.
Excellent points, Chucky. As always happens to me on this issue, I’m on the fence again.
*sigh*
-systemsboy
Chucky-
While you may still see optical on computers in 10 years, I stand by my argument that it is dead for archival purposes.
“Hard drives are actually kinda sucky for archival purposes, due to the need for regularly cycling through concurrent backups.”
I don’t see the need for the averaqe Joe to cycle concurrent backups. Software such as Time Machine and Retrospect take care of the details in the UI while all “Joe” must do is provide a reliable disk to store to which I have mentioned above. The only people I know cycling backups are Enterprise and that is a whole other topic entirely.
“And once solid state takes over, storage form factors and interfaces will change, so the hard drive you’re archiving things on today will no longer be connectable with standard equipment.”
True, form factors and interfaces will change in the future, but at minimum there is a good 3-5 year crossover period in which you will be able to move your backups to the new standard (as I did with IDE to SATA).
Again, I highly stress the point that just because optical (in 10, 20, or 30 years) is available does NOT mean that people will use it for archives. You simply overestimate the time and energy people are willing to devote to backups.
“you can just “burn and forget” with optical discs.”
I think the key word in this phrase is “forget”.
One other item worthy of note:
(Not to mention that the data on it will degrade rather quickly.)
When you take this point into consideration, you must acknowledge the inherent limitations of the average optical disc as well. How many people research disc quality before purchasing discs for archives, general storage, etc. There is a reason that Wal-Mart, Costco, etc do so well: Bang for the Buck. And most (if not all) consumers want the most product for their money. In almost every case, this means quantity, not quality. Therefore “Joe” goes out and buys the 50 discs for $20 since he is going to archive 10 discs every Sunday (yeah, right) for his meager 80 gig media library. Those discs he just purchased have a 2-4 year life span at best before oxidation or warping (because Joe also knows nothing about proper storage of optical either) his archives and he’s wondering why.
I am not saying that optical has no place. You are right, until we are all netbooted, there will be the need for discs. However, software delivery (aside from Operating Systems) will continue to migrate to the cloud negating the need for optical even further which will accelerate the introduction of new ways to install an OS on your system.
Hard Drives, to me, are still the clear winner for Joe. And while I have exercised some diligence on the subject for my own self preservation, I still consider myself a Joe. And I offer my experience to Joes on a daily basis as I try to get people to consider backups.
Although I can’t imagine asking them to devote the best part of a day to it……..
“Hard Drives, to me, are still the clear winner for Joe. And while I have exercised some diligence on the subject for my own self preservation, I still consider myself a Joe. And I offer my experience to Joes on a daily basis as I try to get people to consider backups.”
Look, I essentially agree. I use hard drives for my own archival backup needs.
And if I were advising someone who was doing no backup whatsoever, I’d definitely advise them to pick up a hard drive and use (the awfully executed) Time Machine rather than doing nothing.
But that’s not the entire story.
For one, as Blu-Ray comes down in price, it’s going to become an attractive backup solution for everything other than video.
For two, the expense of doing hard drive backup CORRECTLY starts to get expensive.
In order to replicate the archival robustness I used to have with DVD backup using hard drives, I had to purchase a drive dock, Hudzees, and three hard drives. (If you’re not cycling backups between hard drives, you’re not doing it right. If you’re not keeping an additional backup off-site, you’re not doing it right.) An acceptable level of redundancy is MUCH cheaper to achieve with optical media.
In all, moving from DVD archival storage to hard drive archival storage done properly will cost the average Joe a minimum of about $400 to enter the game (and that’s assuming they are highly educated about how to best purchase their gear.) Not to mention that will be future costs as well.
“The only people I know cycling backups are Enterprise and that is a whole other topic entirely.”
If you’re not cycling your hard drive backups, you WILL lose data at some point. It’s precisely why hard drive backup is sorta sucky: you need to be paranoid about it and continually mange it, or it will end up biting you. Redundancy (including off-site redundancy) is cheap ‘n’ easy with optical media.
Of course, we may be assigning different priorities to long-term preservation of our data. I’m not officially “enterprise”, but I value my data far more than I value my hardware…
“While you may still see optical on computers in 10 years, I stand by my argument that it is dead for archival purposes.”
For the purposes of figuring out if you will be able to easily access your optically stored backups in 10 years, the continued presence of optical drives is the relevant issue. The popularity of optical backup is not particularly relevant.
I did all my backups in the 90′s using DVD-RAM not because I thought DVD-RAM would be a popular archival solution. I did it because I knew I’d likely be able to easily store my archived material and be able to easily access it in 2009. As an end user, that’s what you should be worrying about.
One Trackback
[...] that I intend to archive. That folder used to be called “BurnMe.” But I recently changed my archiving approach such that the “BurnMe” moniker seemed inappropriate. So I renamed “BurnMe” [...]