David begins his rousing critique of the IT industry thusly:

“When people talk about their IT departments, they always talk about the things they’re not allowed to do, the applications they can’t run, and the long time it takes to get anything done. Rigid and inflexible policies that fill the air with animosity. Not to mention the frustrations of speaking different languages. None of this is a good foundation for a sustainable relationship.”

True enough. This is often what people talk about when they talk about IT. They rarely talk about how awesome it is that they have a usable network or rooms full of computers without viruses. But let’s continue.

“If businesses had as many gripes with an external vendor, that vendor would’ve been dropped long ago. But IT departments have endured as a necessary evil. I think those days are coming to an end.”

Typically, businesses don’t have gripes with IT, end-users do. But, okay, I’m curious to hear your reasoning.

“The problem with IT departments seems to be that they’re set up as a forced internal vendor. From the start, they have a monopoly on the ‘computer problem’ – such monopolies have a tendency to produce the customer service you’d expect from the US Postal Service. The IT department has all the power, they’re not going anywhere (at least not in the short term), and their customers are seen as mindless peons. There’s no feedback loop for improvement.”

I don’t think that that’s really the problem with IT departments at all. The problem is that many IT departments make crappy policy decisions that are user-hostile. But that’s not because they have “all the power.” In fact those decisions are often, I’d suspect, borne out of a need to satisfy certain technical goals using limited resources. The characterization that IT departments see their customers “as mindless peons” is offensive to anyone who works in this business, and generalizations such as these do as much to “fill the air with animosity” as any IT policy does. Clearly, the flip-side of “the problem” is an almost willful ignorance on the part of certain members of the tech biz — David, I’m looking at you — to make even the slightest effort to understand what IT departments do before making grand proclamations on the internet about the “end of IT.” While I do agree that there should be better avenues for feedback, that doesn’t mean I can always get what I want. And crying about it is a five-year-old’s tactic.

“Obviously, I can see the other side of the fence as well. IT departments are usually treated as a cost center, just above mail delivery and food service in the corporate pecking order, and never win anything when shit just works, but face the wrath of everyone when THE EXCHANGE SERVER IS DOWN!!!!!”

You’re goddamned right about that. I suspect that a well-respected, well-treated IT department would have warmer, fuzzier feelings for its “customers.” But the fact is that, because people like David continue to see IT departments simply as “cost centers” and not as members of a single team with a shared goal, IT departments continue to be reviled, often by members of the very corporate structures upon which they depend. Unfortunately, this relationship has been sustainable for over twenty years. Probably because, in many institutions, it is a relationship that, though pathalogical in many ways, is necessary.

“At the same time, IT job security is often dependent on making things hard, slow, and complex. If the Exchange Server didn’t require two people to babysit it at all times, that would mean two friends out of work. Of course using hosted Gmail is a bad idea! It’s the same forces and mechanics that slowly turned unions from a force of progress (proper working conditions for all!) to a force of stagnation (only Jack can move the conference chairs, Joe is the only guy who can fix the microphone).”

No, IT job security is not “dependent on making things hard, slow, and complex.” I’m so tired of hearing that. It’s simply not true, and I’d love to hear a concrete, real-world example of some place where that was the case. The fact of the matter is, IT job security is dependent on making things work. Period. If you really think that the IT department uses Exchange Server so that their buddies can get a job, you simply don’t have a clue what IT does.

“But change is coming. Dealing with technology has gone from something only for the techy geeks to something more mainstream. Younger generations get it. Computer savvyness is no longer just for the geek squad.”

Change may be coming. Indeed, I hope it is, because I would love to see the relationship between IT and the end-user improved upon, and, where possible, lessened or even ended. And certainly “dealing with technology” is something everyone has to do these days, but after working in tech education for eleven years, I see no evidence that people have gotten any tech-saavier at all. In fact, from one year to the next, people seem to be pretty much the same: they’re either tech-saavy or they’re not. It has less to do with exposure, more to do with personality. Some people can sing, some people can’t. Making the bad singers listen to music all day doesn’t make them good singers.

“You no longer need a tech person at the office to man ‘the server room.’ Responsibility for keeping the servers running has shifted away from the centralized IT department. Today you can get just about all the services that previously required local expertise from a web site somewhere.”

Apparently, David, you seem to think that all IT does is run servers, which you seem to think requires them to stand next to them inside a server closet somewhere. Hate to break it to you, buddy, but IT does way more than run your shitty-ass fucking servers. IT configures your switches; they deploy your workstations to your labs; they build and maintain your render clusters, your RAIDs your SANs; they provide all your network infrastructure and keep your workstations virus- and botnet-free. And they usually do it from some sunless underground cavern because idiots like you fail to see their importance. You cannot get any of those things from a website.

“The transition won’t happen over night, but it’s long since begun. The companies who feel they can do without an official IT department are growing in number and size. It’s entirely possible to run a 20-man office without ever even considering the need for a computer called “server” somewhere.”

Again, your obsession with servers. And again, I’d love to see some numbers on this. But okay, let’s assume for a minute that you’re right. What you’re basically saying is that there are a lot more smaller companies forming on a regular basis out there. And, sure, smaller companies don’t need an IT department. But smaller companies never needed an IT department. Smaller companies could always outsource their technology needs. That’s not new. That’s not change. That’s just more proof that you don’t know the first thing about what IT departments are or what they do.

“The good news for IT department operators is that they’re not exactly saddled with skills that can’t be used elsewhere. Most auto workers and textile makers would surely envy their impending doom and ask for a swap.”

And that’s proof that you’re a condescending asshole.

Finally, for the straight shit on what IT actually does, John C. Welch says far more than I ever could (as he actually works in IT) and with far fouler language.

UPDATE:
Here’s the inimitable Mr. Welch’s response to the very same article. See? I told ya so. (Thanks, John Mahlman, for the tip.)

It’s Personal
In the Archives post I basically said I’d be archiving all my “non-essential data” to hard drives and reserving optical media archives for only the most essential archives. I should first point out that what I am talking about here is my personal data. This is not necessarily a method I’d use at work or for a client. Archive methods should be specific to the needs of the situation.

The Future
One of my rationales for using hard drives was that hard drives are more likely than optical to be accessible in 10 years with the equipment of the day. It’s this particular idea that received a great deal of criticism, and I’m starting to see why.

Just a few weeks ago I had occasion to archive some museum kiosks that ran from some very old PowerMacs. Luckily, these PowerMacs were just barely of the era when ATA drives were starting to be used as internal drives on Macs. Getting the data off these systems was fairly straightforward. I simply hooked PowerMacs’ the ATA drives up to a firewire case and archived the data to DMG. Shortly thereafter, however, I wanted to perform a similar process with a slightly earlier vintage PowerMac. This machine, however, contained a SCSI drive. And finding a way to access and archive this drive proved almost impossible without going to extreme lengths and making obscure hardware purchases. Had there been some kind of optical archive of these systems, I would have almost certainly been able to pull a backup using today’s equipment.

I’m not sure what the future of optical media is. Until recently, I was pretty convinced it was not long for this world and would surely be displaced as a distribution medium by the web. But after thinking on the comments to that article, and talking to people way smarter than me on such matters, I realize I may be wrong. And if that’s the case, optical will be more likely to be readable than hard drives ten years in the future. But whatever the case, this is certainly true for media from ten years ago. You’re more likely to be able to read ten year old optical media than you are hard drives of that era.

Non-Essential Data
That said, I’d like to clarify the “non-essential data” qualifier I tossed in in the article. To be clear, I’m not completely eschewing optical media for my archives. What the article represented was my shift from optical as my only form of backup to hard drives as a significant if not primary form of data backup and archive.

To get even more specific, in the past I archived everything to optical media. But with the huge amounts of data I now collect, that’s not really so practical anymore, nor is it necessary. So these days the bulk of my data — large, non-essential data, things like ripped DVDs, video captures from tape, software installers, and data with a shelf life (i.e. that is only useful for a period of time or that relies on old versions of software or hardware) etc. — will be archived to hard drive. This will allow easy storage and retrieval. And it should last long enough. The idea is that this data isn’t forever data. It’s stuff I want to keep around for a while, but if I haven’t needed it in ten years, I probably won’t ever need it again.

More important data — of which there’s really not that much, but stuff like big video projects (sans captured media), photos, my websites, contacts, stuff that would really kill me to lose — I’ll be burning to optical. That way I have double backups of it (I’ll also keep it in the hard drive archive), and I’ll have it on a more robust medium that may have a better chance of being readable than hard drives in the future.

So what’s really going on here, for me, is a prioritization of my data backups that’s reflected in my archive procedures. With this prioritization, I can now rely much more heavily on hard drives as an archive medium. Using hard drives I can back up and access a lot more stuff with much greater ease and speed. Doing this allows me to use optical media only for the most important data. But make no mistake: optical will still be an important component in my backup strategy.

Live Archive
I wanted to also take a minute to mention one way hard drives are somewhat future-proof and useful as a true archive, and this is the idea of a live, rolling archive.

In the lab where I used to work we kept — or tried  to keep — a long-term archive of all student work that was accessible to incoming students so that they could look at and benefit from the work of their predecessors. Our students made all sorts of work, from web projects to video and animation projects to installations. And their work was initially being archived to all manner of media, from tape media to optical. There was no standard. By the time I got involved there were projects going back ten or fifteen years, and it was becoming clear that, no matter what medium we used today, we’d need to re-archive everything every so often as data access techniques and hardware evolved. I believe that, in a case like this, where the archive is constantly growing and reaches back well over ten years, but to which access is always required, the concept of the hard-drive-as-archive-medium is a sound one. The implementation would be fairly simple in concept: everything — the entire archive — is kept on a hard drive to which the community has access. As the archive grows, say every few years, it is transferred to larger storage. As storage standards change, it is transferred to the latest greatest medium of the day. Of course, redundant backups are also kept of the entire archive. But since this data is constantly being re-archived, hard drives — or whatever replaces them in the future — make for a sensible way to have a rolling, live archive, and reduce the need for more permanent solutions like optical. Perhaps Chucky, in the comments to Archives, put it best:

“In other words, hard drive archival demands cycling your backups over time to new hard drives with fresh magnetic media and evolving HD interfaces.”

I guess the overarching lesson here, if there is one, is that your archive method should reflect the specifics of your situation; there is no one archive method for everyone. The corollary to that, for me, is that hard drives can (and will) now be a significant part of my archive method.

Then I read it.

It turns out to be an incredibly thorough, yet brief, technical description of the term Cloud Computing. The article does a great job of defining Grid Computing and then uses that definition as point of comparison to arrive at a very real and very useful definition of The Cloud. I’ll be a little less embarrassed to use the term after reading it. Only a little. But still…

Well worth a read for anyone who needs some clarity on the whole Cloud thing.

“You really tell people to reboot for no particular reason?

I don’t believe we should accept that standard from OS X, and what kind of an IT person are you if you’re blindly proposing solutions without any reasoning to back them up?”

What kind, indeed.

I posted a response to the comment that basically explained my position in a nutshell, but I thought it was worth writing a follow-up on the question, both for thoroughness and for those who didn’t happen by the comments section of the previous article, or who may have had a similar reaction.

Reboot!

When a user comes to me with a problem, my primary goal is to fix the problem and get the user working again. Typically what happens is that we have a little discussion about what’s going on. Once I feel I have a good handle on the symptoms, often the next words out of my mouth are, “Did you reboot?” If the answer is no, and the situation permits, I will recommend that they do so. Rebooting is often my first step in troubleshooting.

I believe (though I can’t be completely sure) that my commenter took issue with the approach because blindly telling the user to reboot to solve problems gives the sysadmin no information about what those problems are and what caused them. But in fact, as I’ll demonstrate in a moment, it’s not blind, and it does tell us one important thing: rebooting either solves or doesn’t solve the problem. This in and of itself can be crucial troubleshooting information if there is a deeper problem at work.

But the fact of the matter is that, probably 80% of the time, there is no deeper issue. The fact is that rebooting routinely fixes problems with no other practical solution, such as the one I described in my article. Moreover, it provides the end-user with a method of troubleshooting that is likely to achieve the desired results — allowing them to get the system or an application back to a working condition — without the need for admin intervention. This is win-win: it saves both the user and me time and energy and, by determining whether or not a reboot is helpful, still provides valuable troubleshooting information.

I would even argue that rebooting should almost always be the first step in troubleshooting. When a user comes to me with a problem, I have no idea what they’ve been doing on that system. I have no idea how many nor which applications they have open. I have no idea what sorts of preferences they’ve set. There’s simply no way for me to reliably predict the state the user has put their machine in, and thus whether or not this is a system- or user-level problem. The only way for me to get things back to some semblance of a known, working state is to reboot the system. Rebooting has myriad benefits, not the least of which are: clearing stale caches; recreating network connections; and freeing up RAM and disk space. In fact, it seems almost crazy to proceed with most troubleshooting without first rebooting.

You may have noticed that I keep saying that rebooting should almost always be the first troubleshooting step. That almost is there because, obviously, there are times when rebooting is not a good first step. Primarily, when a user stands to lose work by rebooting. If an application is hung and the user hasn’t saved his document, for instance, I don’t tell them to reboot. Rebooting would be bad in this instance. Also, I usually try other troubleshooting methods first on my own systems, with which I am better acquainted (though this is often to my own detriment and rebooting would have been quicker and easier, as happened with the Final Cut bug).  Another instance in which I avoid a reboot is when there is a persistent problem that is not solved, or is only temporarily solved, by a reboot. Then I do need to get on that system and attempt to understand a problem. And that’s what I do.

But, because Mac OS X is very reliable on the whole, these instances are extremely rare in my experience. The majority of problems are minor and are easily and permanently rectified by a simple reboot. I stand behind that recommendation, and any search of Mac troubleshooting articles will reveal that the advice is almost universal. That’s because it works.

So, hopefully it’s clear by now that I’m not “blindly proposing solutions without any reasoning to back them up.” Hopefully it’s clear now that there are a lot of good reasons to try rebooting as a first troubleshooting step.

And hopefully it’s clear that the kind of sysadmin I am is the kind that likes to get his users back up and running again with a minimum of friction.

Let’s start with a rundown of the existing approaches.

Roaming Profiles
The approach Windows computers use is called Roaming Profiles. The way Roaming Profiles work is pretty simple. Users’ home account data is stored on a centralizd server. When the user logs in to a client system her data is downloaded from the server to the client machine. She will access her data locally for the duration of the session. When she logs out the data will be synced back up to the server. The advantage of this approach is that the user has local access to her data and isn’t beholden to the network while actually working. This makes data access generally faster and more reliable. The big disadvantage here is that if the user makes any big changes or creates any big files, a large data transfer will happen at log out, and then again at login to subsequent machines that aren’t yet synced to the server. This both slows down the login/logout process and places an often undue burden on the network.

Because of the sorts of environments I tend to work in — data-intensive, video and image oriented facilities that create a lot of data — my experience with Roaming Profiles has been fairly poor. For my uses they’ve required a lot of management and have been somewhat unreliable. But, for the purpose of maintaining a user environment across multiple networked systems, they work well enough if you understand and plan for their inherent limitations.

Network Home Accounts
The method used by *NIX systems, Mac OS X included, for time in memorial, is generally referred to these days as Network Home Accounts. In the Network Home Account model, as with Roaming Profiles, the user’s home account data is stored on a server. But when the user logs in using Network Home Accounts no data transfer occurs. Instead, the home account data is accessed directly from the server: new files are written directly to the server; settings files are read directly from the server; everything happens over the network and the network share that contains the user’s home account data is treated just like a local volume. The speed advantage over Roaming Profiles at login and logout is obvious; there’s simply no lag time as data gets transferred between the client and the server, because there simply is no data transfer. On the other hand, accessing your entire home account over the network can be slower than a local account even on the speediest of networks. And on slower networks, or networks with a great deal of traffic, you’ll definitely notice the slowdown. There are also potential problems due to the constant reliance on the network and server. If the network becomes congested or the share becomes unavailable even for a second you’re liable to feel the pain. If either goes down you’re dead in the water until they’ve returned to service.

As network home account models go, I like this one the best. I’ve used it a great deal in educational settings in which resources are almost completely shared and it’s fairly reliable and usable. But even this model can be frustrating and is less than ideal when compared to working from a local home account.

Portable Home Directories
The final model is called Portable Home Directories. Devised by Apple for laptop computers with occasional — but not constant — access to the network hosting home account data, Portable Home Directories attempts to combine the best of the Roaming Profile and Network Home models by providing finer-grained control over the sync process in what is otherwise a Roaming Profile approach. So, Portable Homes sync to specific data at specified times when they’re on the network. Fine-grained control over what is synced and when is intended to mitigate performance issues at login and logout.

My main problem with this approach is that, in my admittedly limited tests, it doesn’t seem to work very well. I also don’t like the level of management required. The other models, once set, require little if any tweaking whatsoever. But I could see spending a great deal of time and effort getting my Portable Home Directory settings just so.

The Problem
But my overarching beef with all these models is that they don’t really jive with the way most people in most of the environments I’ve encountered actually use their computers. This makes them use system resources less efficiently and yields a poorer user experience than if they did.

So how do most people work? Well, what I’ve tended to see in the media-based environments in which I’ve worked is that users are generally assigned a single computer. It’s this computer from which they work almost all the time. Indeed, this is how I work in my current job. I’m almost always working from the computer in my cubicle. Almost.

Every now and then, however, I need to work from a different machine, and there are often times when I’m doing this that I realize that it would be extremely handy to have my entire home account — all my environment settings, files and folders — available to me on this other machine. But I don’t. They’re over there, on my cubicle machine. If only I could use the home account on my main computer directly, as thought it were a Network Home Account.

And this is the basic idea behind Satellite Home Accounts.

Satellite Home Directories
All the current models rely on the user’s data being stored on and accessed from a centralized server. But why? Why can’t the server be the user’s main computer? In the Satellite Home Account model, the user’s primary computer becomes the home account server for any user that sets her account as a Satellite Home Directory.

The way I envision it, it would actually be quite simple to set up. In the Accounts preference for the user would a be a tickbox to activate Satellite Home Directories. Once activated, the user’s system would begin broadcasting Satellite Home Directory information, just like Mac OS X broadcasts Network Home Account info. The user would then work locally as normal, but when logging into another system on the network — a system that’s listening for SHDs — the user would be presented with her home account over the network, shared directly from her primary system rather than from a centralized server. Simple.

Among the great benefits of this system are its simplicity and the fact that it requires no server. But the chief advantage comes from the fact that the Satellite Home Directory system works the way users tend to work. When you’re on your main computer, which you are 99% of the time, you get a fast, responsive, local home account. When you move temporarily to another system, your environment follows you. It’s a bit slower, sure. But hey, it’s only temporary. The network overhead is significantly reduced from the other methods, and the user experience is also enhanced. It’s win-win.

There’s certainly no technical reason an implementation like this would be impossible or even particularly difficult. Most of the technology already exists, either in Mac OS X client or Server. All we need is for someone to program it. And while I doubt there’s likely much interest on Apple’s part to build something like this, I really think it’d be damn sweet.

And a boy can dream, can’t he?

Final Cut 7 Garbled Image Display

This is my dog:

Garbled Dog

She is not normally purple and green and swirly colored.

After an hour or so mucking about, reinstalling the application, trashing prefs and otherwise performing the usual maneuvers, I decided to take my own medicine and reboot. Why I don’t just always do this first — like I tell everyone else to do — is beyond me. But sure enough, it worked.

Normal Dog

Ah! That’s better! Crazy mutt!

Some History
In the very cross-platform lab where I used to work we were continually on the hunt for the best filesystem solution for users of multiple platforms when they were using external firewire or USB drives. That is, some folks wanted their drives to be accessible from both the Mac OS and Windows. On the surface this can seem like an easy problem to solve — Fat32 (or “MS-DOS” as it’s called in Disk Utility) is readable and writable on both platforms. But it’s not so cut and dry.

The biggest problem for me was video. See, I taught — and continue to teach — a video class in that very same department. We use Final Cut Pro as our editing software, for a variety of reasons, not the least of which is the fact that I prefer to work on the Mac. I require my students to have a firewire drive appropriate to showing in-progress video work in class. But Fat32 has a 4GB file size limit, and video captures can often exceed that limit. What happens when this limit is exceeded is interesting from a systems standpoint, but devastating from a user standpoint.

Video and Fat32
When capturing video in Final Cut Pro to a Fat32 volume, what happens is that the video file gets segmented. That is, the capture file gets written in 4GB chunks. Initially, Final Cut will see these chunks and understand what they are. But after saving the project and quitting the app Final Cut will no longer be able to locate the captured media because it’s in multiple files with different names. The path to the media that FCP relies on is now, essentially, broken. This actually happened to a student of mine some time ago, and we were able to use the cat command to reconstruct the single movie file onto an HFS+ volume and then point FCP at the reconstructed file. Boy was that fun.

NTFS
We’ve often looked to the ever-popular NTFS file system as a possible future solution. It does not have such small file size limits, and it’s readable on Mac and Windows. But the Mac has never been able to write to NTFS. So, in the past, our solution in the lab — our recommendation for users who really needed a dual-format drive with read/write capabilities on Mac and Windows — was to use the HFS+ filesystem on the drive and use MacDrive on Windows to read and write to that drive. Inelegant? Yes. But it mostly worked.

Mac and Windows Partitions
Another potentially attractive alternative to a single, dual-platform volume was the idea of splitting the drive into two partitions and dedicating each partition to a platform/filesystem. This way, even if all your Mac and Windows data wasn’t all mushed together in one volume, you could at least keep it all on one device. This solution would likely work for the vast majority of users. Unfortunately, there was never a particularly straightforward way of doing this. Sure, it was doable. But it wasn’t easy, and it wasn’t something you could tell new students to do. In fact, it was likely to require admin access and command-line heroics, and so just wasn’t a viable solution to anyone but the most die-hard user. Until now.

Without too much mucking around, it’s now possible to create a dual-format external drive that contains a mac-formatted partition and a Windows-formatted one.

MacFUSE and NTFS for Mac OS X
The first step is the only really tricky part, and it’s not even that tricky. If you have need for a dual-format drive, this should be pretty easy for you. You’re going to need to install the MacFUSE and NTFS packages. In a nutshell, MacFUSE is an experimental set of tools for doing unsupported things with filesystems like SSH, FTP and, of course, NTFS on your Mac. And, experimental though it may be, I’ve been using it for quite a while and have not had any problems to speak of. Installing MacFUSE and the NTFS drivers will allow you to mount NTFS volumes with read-write access.

MacFUSE

So, if NTFS can be mounted read-write on the Mac with MacFUSE, and it’s obviously read-write on Windows, and it doesn’t suffer from the file size limitations of Fat32, why not just use NTFS as your über-filesystem and format the whole drive with it? That’s a great question, and I’m glad I asked it!

The thing about getting NTFS read-write access on a Mac with MaFUSE is that it’s very much a hack. Yes, it works, but it has its problems. First and foremost among them is the fact that Final Cut Pro is really not a fan. In fact, FCP might just be the best barometer of a good cross-platform solution as it seems to be so picky about filesystems. So far, the only filesystem I’ve seen work consistently well with Final Cut is HFS+. No surprise there. And on NTFS it gets downright crazy. Files sometimes won’t open. Sometimes they won’t save. It’s a scary mess, and I wouldn’t trust my FCP data on NTFS for any amount of money.

But, what the MacFUSE NTFS package does get you is a relatively easy way to format your drive with separate Mac and Windows partitions, and this, at least in my tests seems to work just fine.

NTFS-3G for Mac OS X

The easiest way to get everything you need is to go to the NTFS-3G for Mac OS X website and download the latest package. This package will install the most recent non-beta version of MacFUSE as well as the latest NTFS libraries, and contains everything you need. Once you’ve installed this bundle, you’ll need to reboot your system.

Creating the Dual-Partition Drive
After the reboot you’ll see a new filesystem option when you go to format drives in Disk Utility.

A New Option

Moreover, that option will be available to individual partitions of drives that are otherwise formatted. And that’s what’s new (to me) and what allows the magic to happen. Here’s how you do it.

1. First, if you have any data on the drive that you need to preserve, back it up. This process WILL ERASE YOUR HARD DRIVE.
2. Next, select the drive you want to dual-format and choose the Partition tab.
3. Select a Volume Scheme. I’m just doing the simplest, two-partition scheme, with one Mac and one Windows partition, but you can certainly get more Byzantine with it if you’d like.
   

   Volume Scheme

4. Set the Format for the partition you want to use on the Mac to “Mac OS Extended (Journaled),” give it a name and a size.
   

   Mac Partition

5. Set the Format for the partition you want to use on Windows to “Windows NT Filesystem (NTFS-3G),” give it a name and size.
   

   Windows Partition

6. Under the Options… set the partition scheme to “Master Boot Record.” This is needed for Windows to see your drive.
   

   Partition Scheme

7. Finally, hit the Apply button. You’ll be warned that everything is about to be deleted. Click through, and after a few seconds you will have completed the formatting process and your dual-format drive will be ready for use on Mac and Windows.

Caveats
As I said, so far this has been working really well for my class. You may still want to file it under “experimental” for the time being, at least until you’re sure it’s working safely. But I’m confident enough in this method to recommend it to my video students who also need some external Windows drive love.

It’s also important to keep in mind here that I am not endorsing using the NTFS partition for Mac data of any kind. Doing so is surely unsupported by Apple, and by all reports is fraught with problems.

The other thing to keep in mind is that, unlike with a GUID partition table, you will not be able to resize or split partitions without completely erasing the drive.

Erases Everything

Conclusion
Lastly, I realize that this process is hardly new, nor am I the first to discover it. It was pointed out to me by one of my video students, and I have a feeling the new admins at my old job have been using it for some time. But it’s new to me. This is the first I’ve heard of this and it’s exciting to me from an academic standpoint, in the context of my old job, in the context of my class, as a new option I can offer to whomever might need it, and as a symbol of progress — however small or kludgy — towards cross-platform filesystem solutions. This is just another of the very cool advances made possible by the existence of the MacFUSE (and the original Linux FUSE) effort. It’s very cool to see this sort of thing coming to fruition at last!

Another intriguing extension of the MacFUSE project — and one that I’ve used a bit myself — is MacFusion, which allows for mounting of data over network protocols such as FTP and SSH. I’m sure there are tons of others. I highly recommend folks — particularly SysAdmins — check out and familiarize themselves with MacFUSE in general, as well. As much as has been done since the last time I looked at it, there is still a ton of future potential in the project, and I see it increasingly becoming a part of the admin’s toolbox.

This isn’t working for me anymore.

Optical media is no fun to use for backups. It’s slow, requires keeping a stock of media on hand, and it’s often undersized for today’s hefty data needs. For instance, backing up my current BurnMe folder will require about fourteen DVD-R discs. Burning and verifying those DVDs will take as many if not more hours. And then it will all need to be cataloged somehow, which is also a lengthy process. Finally, retrieval is almost as slow and tedious: find the disc you need in the catalog, get it from storage, load the disc up and copy the data back to the computer, which can also take quite some time. And if, heaven help you, there’s one tiny scratch on that disc you could lose all that data.

So I’m switching to a better way: hard drives.

One of the best bangs for your buck per-gigabyte of storage is, of course, the hard drive. In addition to being cost-effective, they’re fast, they hold a lot of data, they’re read-write and extremely versatile. You can use them internally, depending on the drive and computer in question, of course. Or you can use them externally in any number of ways, the most obvious being in a firewire or USB case. Using such a case allows for the attachment of the drive to just about any computer you can get your hands on. Drives don’t require much physical space, and because they’re so fast, they’re quick and easy to catalog and restore from. In fact, using hard drives as an archive solution entails little difference from accessing local storage. The hardest part is getting the drive off the shelf and getting it attached to your machine. But there are hardware solutions to simplify that process as well. Hard drives, which have largely remained the same since I began working in this business ten years ago, are more future-proof than optical media, which is constantly the subject of format wars and which is often subject to a raft of compatibility issues. Put another way: getting data off a hard drive in the next ten years is more likely to be supported by my current hardware than getting it from a DVD.

So, my plan, going forward, is to use hard drives to archive all non-essential data. I will continue to archive certain critical files to DVD in addition to the hard drive archive. But most of my stuff will be on hard drives. Things like bittorrent backups and movie files, old audio, video and web projects, images and what have you will stay on hard drives earmarked for the archive. Once one of those drives is full and I’m no longer using the data, I’ll burn anything critical — finished projects and their assets, for instance — to optical media, catalog the drive with CD Finder or similar software, and then put the drive in a Hudzee and up on the shelf. Done and done!

This greatly speeds and simplifies my archive procedure. And, because of the increased space and time efficiency, I can archive a lot of data I would have thrown out in the past. I still have yet to work out the details of this system, but I think it will be a vast improvement over the old burn-and-catalog weekends of my not-so-distant past.

If anyone has thoughts on how best to archive data I’d love to hear them in the comments.

It really is somewhat amazing that the preferred method for installing a good deal of Mac software — the so-called “Manual Install” — is one that’s liable to be confounding to so many users, particularly given that Apple has gone to such great lengths to simplify software installation. And I agree with much of what I’ve read on at least one point: The problem is the DMG.

“They drag the application to their dock directly.

This creates a link to the file inside the disk image, which means that every time they try starting Firefox, the disk image is unpacked and mounted, and starting of Firefox becomes very slow, which makes it a bad experience.“

It’s true; I’ve seen very computer-savvy users constantly launch Firefox directly from a Dock icon that’s linked to the item on the disk image. And there are certainly many users to whom you’ll never be able to explain exactly what a disk image is, how it works or why it exists. Frankly, as useful as it is to geeks like you and me, it’s kind of a crazy concept for anyone who doesn’t care about these sorts of things. And I’m of a mind that most users don’t and shouldn’t.

That said, our options are limited when it comes to installing software. As Limi points out, most apps these days are actually folders (another conceit most users aren’t and shouldn’t be aware of), and so they must be installed from some sort of container, particularly if they’re coming from the Internet. This can be either a ZIP file or, as is generally the way on the Mac, a DMG. Typically, Apple uses a DMG that contains an Installer package for distributing their own software, probably because their stuff is usually not simple drag-and-drop stuff. They’ve also created a special “internet-enabled” flag for DMGs that, when applied, will be recognized by Safari, which will proceed to open such DMGs and run any installers contained therein. This, at least, gives us a way to accomplish some of Limi’s stated goals for the Firefox installer:

1. Start the Firefox download.
2. When the download is complete, the disk image will mount automatically (if they were using Safari), and the Firefox installer runs.
3. The install procedure continues similar to how it happens on Windows.
4. As the last step of the process, the installer lets you set Firefox as the default browser, and start the application immediately. We have seen users forget that they just installed Firefox if you don’t let them start it at the end of the process, and make that the default choice.

Having some experience with Mac OS X package creation, I decided to see if I could quickly whip something up that met these goals. I believe I have been successful. Here’s the installation procedure I’ve designed:

1. The user downloads the Firefox Installer from Safari.
   

2. Safari Opens the DMG and runs the Installer.
   

3. The Installer uses a standard method by default, which requires a minimum of clicks, and installs Firefox in the default location, /Applications.
   

4. Upon pressing “Install,” the Installer requests authentication.
   

5. Before proceeding, the Installer’s preflight routine quits Firefox if it is already running (this is not necessary, we do it as a precaution).
   

6. The Installer then proceeds with the installation of the app.
   

7. When it has finished it will open the containing folder and highlight Firefox.
   

8. And then it will launch the app. If this is the first launch, Firefox will ask the user if she wants to set Firefox as the default.
   

This is the basic install procedure, but some more advanced users will, of course, want to customize the install location of the app. To that end, the installer I’ve built also contains a “Customize” option.

Clicking this will allow the user to customize the install location of the app by selecting “Other” from the Location pull-down.

And then choosing the location from the resulting menu.

I believe this method offers the best of all possible worlds given the current state of off-the-shelf installation options for Mac OS X. It offers some level of customization for advanced users, while still offering a guided experience for less Mac- or tech-savvy users. And I believe it fulfills most or all of Limi’s stated goals for a Firefox installer. There are even numerous things we can do to further customize this guided experience — things like adding graphics and explanatory text, or running additional pre- and post-flight scripts to perform certain behaviors.

And this will work great for most folks who use Safari as well. But for anything beyond this particular scenario — i.e., anyone trying to install apps from browsers other than Safari, which would be anyone who uses the above method, i.e. Firefox users as well as users of any other browsers — we still have a problem. That problem is DMG behavior. The DMG, despite being immensely useful and a perfectly good application and installer container, once downloaded, is easily forgotten. It’s behavior and purpose are arcane to most users. They don’t know what to do with it and are confused by its presence. And it has nothing to do with anything normal users think of as “Software Installation.”

Some developers have rolled their own solutions to some of these problems. But I think the answer really needs to come from Apple in the form of a unified, internet-savvy Installer format. Something that knows where it comes from and what it’s supposed to do, and once downloaded, just does it. Something that developers can all use for any kind of installation, even so-called “simple drag-and-drop.” Something that just works, dare I say, at least as well as it does on Windows. Preferably better.

Given the level of abstraction of so many concepts in modern operating systems — applications are folders, disk images are files that can represent volumes, Soylent Green is People — I think drag-and-drop installers are bound to be confusing to many users. The guided experience, while a bit of a bummer for very advanced users, should be the preferred method. And I think the best candidate for improving upon that method is ultimately Apple.

One thing that would help the situation immensely, though, is if browser developers made their apps internet-enabled-DMG-aware, like Safari is. This goes a long way towards mitigating the confusion wrought by the ever-confusing, yet — at least at this point in the game — fairly necessary DMG.

[Gazes dreamily off into space for a moment. Then abruptly snaps to.]

The trigger for this failure, ironically, was our attempt to make a backup. (Oh, technology gods, thou art a riot!) See, our original goal was to update the OS to Leopard, but with all the craziness going on these days we decided to clone the drive before we proceeded with said update. But in the course of cloning, it would appear in retrospect, we hit a bad block and triggered the first of what would be many, many disk errors. Unable to pull a backup, we began our descent into drive repair hell in our latest heroic attempt to salvage that ever-important thing contained on and lost from drives: the data.

File-Level Attempts
Our first try was with Disk Utility, which consistently reported, in all red text, that it could neither verify nor repair the file system. Right. On to attempt number two.

Disk Warrior is my go-to utility for any sort of file system damage that Disk Utility is unable to repair. I’ve rarely seen a disk that one of these two apps couldn’t fix. Today would be one of those rare days. After mounting the drive on a known good system using Target Disk Mode, we let Disk Warrior perform its initial scan of the drive. What we found was decidedly ugly. Disk Warrior told us that it was unable to replace the borked directory with its shiny new, replacement directory because of a “disk malfunction.”

That’s when we knew the drive was fried.

Disk Warrior Report: Bad News

When a hard drive has problems, 99% of the time those problems are directory related. That is, the hard drive contains data about the files on disk — where they belong, how many there are, how the disk is partitioned and so on. And usually, when there is a problem with a drive, it is because this information has been corrupted somehow. These days there are numerous utilities that can easily and accurately repair these sorts of problems, Apple’s included Disk Utility among them. Sometimes the damage is too extensive, though, so we turn to something a bit more drastic, like Disk Warrior. Disk Warrior forgoes the repair, and instead scans the disk and creates a brand-spankin’ new directory, replacing the broken one with its new one once you’ve made sure everything is cool, and perhaps made a backup. Now, when Disk Warrior is unable to do this it’s indicative of a much more serious problem. When this happens it is very likely that the drive hardware is beginning to fail.

Time for a new drive.

What Disk Warrior does in these instances is it shows you the best picture it can muster of the drive’s contents in a read-only preview, and then advises you to backup as much as you can before total failure. So that’s what we did. You’re never sure how much time you have in these situations, so we went through folder by folder trying to locate and backup the most important files first. With each successive copy the drive became slower and slower. Luckily, we were able to pull the most recent, most important files. Most everything else was backed up or able to be easily reconstructed.

Block-Level Attempts
Once we had gotten the most important stuff we decided to see what else we could get. I tried running some rsync commands and got some stuff that way, but not much, and it was taking forever. Once I’d given up trying things at the file level, I decided to make my last ditch effort with a well-worn but powerful little UNIX command called, simply, dd. (No, it does not stand for “Drives Die,” though maybe it should.)

The dd command reads data from a disk at the block level and copies it from standard input to standard output which can then be written to a file of your choosing. I use dd by running it on the /dev entry of the drive in question and writing the output to a disk image file (DMG):

sudo dd bs=512 if=/dev/disk3s3 of=/Volumes/Work/LastDitch-DD-01.dmg conv=noerror,sync

The good thing about dd is that you can instruct it to skip damaged sections of the disk. That’s what the “noerror” option is for. The downside to dd is that it wants to read the entire disk, and that makes it very slow. In this instance I was not able to rescue any data, mainly because, as I soon discovered from my dd runs, the disk was just too far gone. I did learn some interesting strategies for using dd to recover data though.

The first thing you can try if dd is running slowly is to increase the block size. This is how much data dd will consider before moving to the next read. The default is 512 bytes. I’ve read upping that to 51200 will sometimes yield speedier results:

sudo dd bs=51200 if=/dev/disk3s3 of=/Volumes/Work/LastDitch-DD-02.dmg conv=noerror,sync

In my case it did not, primarily, I believe, because there was a problem in the beginning of the drive, and dd was having trouble moving past that spot. So another thing you can tell dd to do is to skip a certain portion of the drive, say the first 2 GBs:

sudo dd bs=51200 if=/dev/disk3s3 skip=2000000 of=/Volumes/Work/LastDitch-DD-03.dmg conv=noerror,sync

Finally, you can also tell dd to only write in 1 GB chunks, using the count option:

sudo dd bs=51200 if=/dev/disk3s3 count=1000000 skip=2000000 of=/Volumes/Work/LastDitch-DD-03.dmg conv=noerror,sync

I was getting some good results after having skipped the first 2 GBs — apparently they were really damaged — so I decided to write a script that would skip the first 2 GBs and then begin writing out 1 GB chunks of data. It would’ve looked something like this:

sudo dd bs=51200 if=/dev/disk3s3 count=1000000 skip=2000000 of=/Volumes/Work/LastDitch-DD-Chunck-01.dmg conv=noerror,sync
sudo dd bs=51200 if=/dev/disk3s3 count=1000000 skip=3000000 of=/Volumes/Work/LastDitch-DD-Chunck-02.dmg conv=noerror,sync
sudo dd bs=51200 if=/dev/disk3s3 count=1000000 skip=4000000 of=/Volumes/Work/LastDitch-DD-Chunck-03.dmg conv=noerror,sync
...

Etc, etc, up to the 40 GBs needed to scour the drive. I never got to write the script, though, because the last dd command seized up and the drive began making the clicking, knocking and whirring sounds of its agonized and tortured death. It was quickly dying. We could do no more.

At this point, mainly for my own edification, I decided to see what could be done outside the confines of my home office. I decided to get a quote from Drive Savers.

Hardware-Level Attempt
Drive Savers, perhaps wisely, does not list prices for their services on their website. To get an estimate you have to give them a call. When I called them I was greeted by a very friendly and helpful service person — yes, person — which was really nice. The last thing you want to deal with when you’re having a mechanical failure is a machine. The person on the other end of the line asked me a few basic questions to gauge what state the drive was currently in, things like what attempts I had made to rescue the data, would the drive mount, and the like. After entering this info into her systems she directed me the “Tips, Techniques and Solutions” page on their website (very useful — love the drive sound audio samples), stressing above all that in order to have the best chance of recovery at this point the drive should not be powered on again. She also offered up some information about the company and what they do: For one, they started with Mac data recovery and are an all-Mac shop, which surprised me a little. She also pointed me to information on the Drive Savers clean room, a vital part of data recovery at the hardware level. She then took my email and contact info and gave me both a written and verbal estimate of how much I could expect to spend should I decide to go ahead and have Drive Savers attempt to save my data (I don’t think they’ll actually save the drive). All in all it was a very pleasant and informative experience. Normally I am loathe to use the phone for business, but Drive Savers really seems to know what they’re doing, at least when it comes to pre-sales customer service, and that counts for a lot in my book.

This is, of course, all prep for the fact that, if you do want to make the attempt at data recovery, you’ll be expected to drop a significant amount of money. This is hardly surprising. Those clean rooms don’t look particularly cheap to build or maintain. And if data recovery at the hardware level is anything like it is at the software level, it is a laborious and time consuming process. I was given a range of prices ($500-$2700 dollars) and told that the cheapest I could expect to get away with — the economy plan, which isn’t as fast as some of the other, more expensive plans — was $500 dollars. But it was likely I’d pay somewhere closer to the upper third of the range, more like $1500 to $2000 dollars. It all depended, of course, on how much data Drive Savers could recover.

I didn’t really find these prices particularly surprising. I’d long heard how much such a recovery could cost. That it would be pricey. I was glad that I was not in a situation that required me to fork out this amount of money. I’m glad such a service exists for the odd catastrophe, though I hope never to have to use it. Drive Savers’ website offers advice on keeping backups:

“Backup strategies:
* Invest in redundant backup systems
* Establish a structured backup procedure to make copies of all critical data files, using software compatible with the operating system and applications
* Periodically test the backups to verify that data, especially databases and other critical files, are being backed up properly
* Keep at least one verified copy of critical data offsite”

Sage advice, all. Take it from those who know all too well.

The Belly of the Beast
Once we’d decided not to use a hardware data recovery service the only thing left to do was spec out, buy and install a new hard drive. This wasn’t terribly difficult, but as is so often the case, there was the odd snag or two.

Before we even bought a drive, I wanted to see how hard it would be to open the PowerBook for servicing. If it was going to be a bear — and some PowerBooks are certainly easier to crack than others — I’d let the fine technicians at Tekserve do the job. So I went in search of manuals and instructions for this particular model of PowerBook. Without too much trouble I was able to locate, at Apple’s site, the manual for our 1.67 MHz, 15″ Aluminum PowerBook. It contained no instructions for hard drive replacement, which is generally a sign that Apple would rather you not attempt the repair yourself. That got me a little worried.

Finally, however, I found instructions — great instructions, no less — at the venerable — awesome, actually iFixit.com. iFixit, for those of you who don’t know, provides step-by-step, illustrated guides on taking apart and performing repairs on Apple hardware. For free. They’re amazing. I feel guilty not buying anything from their site. Oh yeah, they also sell parts, tools and service as well. I love them. And from what I could see, the repair would be tedious — lots of screws — and would require a trip to the hardware store — blasted tiny hex screws! — but it would be doable. Still, taking things one step at a time, I thought I’d perform the teardown before buying the drive. Just in case.

And perform I did. Using iFixit’s excellent guide, I was able to crack the PowerBook in short order. I was ready to buy a drive.

Buying a Drive
There are two things SysAdmins typically are, particularly when it comes to technology: cheap and lazy. Hunting for a replacement drive brought both of these qualities in my personality to bear. I was looking for the cheapest replacement I could find, at the location closest to my house, a SysAdmin’s dream hunt. The closest proper computer tech shop to me is Tekserve, with Best Buy a close second. Tekserve doesn’t list what bare drives they carry, if any. But Best Buy seems to have the goods. But Best Buy is still a good half hour train ride, so I did some physical recon at my nearest Radio Shack, which happens to be right around the corner. They informed me that, though they did not have any bare drives in stock, they did have portable USB drive on sale. Drives from which I could pull and the internal component and install it in the now drive-less PowerBook. In fact, they had a 160 GB Iomega Prestige for less than a bare drive would have run me at Best Buy — a mere $75 clams post-sales-tax. Not bad. I took it.

I’d like to pause here and see if anyone can guess why this didn’t work out for me. You have pretty much all the data you need in this article to figure it out. But don’t feel bad if you can’t. The good lord knows I surely didn’t. I’ll wait a minute… Pretend there’s Jeopardy countdown music playing… Aaand…

Okay. Did you guess it?

I got the drive home, popped it out of its case and went to put it in the open PowerBook. But it didn’t fit. (Have you guessed it yet?) Here’s the thing: PowerBooks use 2.5″ ATA drives (Parallel ATA, or PATA), but drives in today’s externals are all now SATA (Serial ATA) drives. Blast!

Oh well. At least it was cheap.

Another quick look at the web revealed that all the bare drives at Best Buy were SATA as well. Blast again!

The nearest ATA drive I could find was at J&R, which is all the way downtown, almost at the very tippy-tip of Manhattan — far. So that’s where we went.

Once we got back, we installed the drive and — the very first thing to go right all day — it worked. Perfectly. Things were finally looking up.

Once we had installed the drive it was simply a matter of formatting it, installing the latest version of Leopard (which is all we ever wanted to do in the first place) and copying over the rescued and reconstructed data. Oh, did I mention that the reason the client wanted Leopard was for Time Machine? Yup. Backups. Great timing. So we set up Time Machine as well. All that went exceedingly smoothly and our repair is, at last, complete. Whew! What an ordeal!

But, man, did I ever learn a lot.

The Life and Death of Hard Drives
So yes, drives die. How they die, though, is almost as important as how they lived, and certainly as interesting. It’s somewhat comforting to know that this drive, while quite dead indeed, did not die in vain. Rarely have I had the opportunity to learn so much about practical drive recovery. I have that PowerBook drive — specifically its death, in fact — to thank for my lesson.