[Gazes dreamily off into space for a moment. Then abruptly snaps to.]

The trigger for this failure, ironically, was our attempt to make a backup. (Oh, technology gods, thou art a riot!) See, our original goal was to update the OS to Leopard, but with all the craziness going on these days we decided to clone the drive before we proceeded with said update. But in the course of cloning, it would appear in retrospect, we hit a bad block and triggered the first of what would be many, many disk errors. Unable to pull a backup, we began our descent into drive repair hell in our latest heroic attempt to salvage that ever-important thing contained on and lost from drives: the data.

File-Level Attempts
Our first try was with Disk Utility, which consistently reported, in all red text, that it could neither verify nor repair the file system. Right. On to attempt number two.

Disk Warrior is my go-to utility for any sort of file system damage that Disk Utility is unable to repair. I’ve rarely seen a disk that one of these two apps couldn’t fix. Today would be one of those rare days. After mounting the drive on a known good system using Target Disk Mode, we let Disk Warrior perform its initial scan of the drive. What we found was decidedly ugly. Disk Warrior told us that it was unable to replace the borked directory with its shiny new, replacement directory because of a “disk malfunction.”

That’s when we knew the drive was fried.

Disk Warrior Report: Bad News

When a hard drive has problems, 99% of the time those problems are directory related. That is, the hard drive contains data about the files on disk — where they belong, how many there are, how the disk is partitioned and so on. And usually, when there is a problem with a drive, it is because this information has been corrupted somehow. These days there are numerous utilities that can easily and accurately repair these sorts of problems, Apple’s included Disk Utility among them. Sometimes the damage is too extensive, though, so we turn to something a bit more drastic, like Disk Warrior. Disk Warrior forgoes the repair, and instead scans the disk and creates a brand-spankin’ new directory, replacing the broken one with its new one once you’ve made sure everything is cool, and perhaps made a backup. Now, when Disk Warrior is unable to do this it’s indicative of a much more serious problem. When this happens it is very likely that the drive hardware is beginning to fail.

Time for a new drive.

What Disk Warrior does in these instances is it shows you the best picture it can muster of the drive’s contents in a read-only preview, and then advises you to backup as much as you can before total failure. So that’s what we did. You’re never sure how much time you have in these situations, so we went through folder by folder trying to locate and backup the most important files first. With each successive copy the drive became slower and slower. Luckily, we were able to pull the most recent, most important files. Most everything else was backed up or able to be easily reconstructed.

Block-Level Attempts
Once we had gotten the most important stuff we decided to see what else we could get. I tried running some rsync commands and got some stuff that way, but not much, and it was taking forever. Once I’d given up trying things at the file level, I decided to make my last ditch effort with a well-worn but powerful little UNIX command called, simply, dd. (No, it does not stand for “Drives Die,” though maybe it should.)

The dd command reads data from a disk at the block level and copies it from standard input to standard output which can then be written to a file of your choosing. I use dd by running it on the /dev entry of the drive in question and writing the output to a disk image file (DMG):

sudo dd bs=512 if=/dev/disk3s3 of=/Volumes/Work/LastDitch-DD-01.dmg conv=noerror,sync

The good thing about dd is that you can instruct it to skip damaged sections of the disk. That’s what the “noerror” option is for. The downside to dd is that it wants to read the entire disk, and that makes it very slow. In this instance I was not able to rescue any data, mainly because, as I soon discovered from my dd runs, the disk was just too far gone. I did learn some interesting strategies for using dd to recover data though.

The first thing you can try if dd is running slowly is to increase the block size. This is how much data dd will consider before moving to the next read. The default is 512 bytes. I’ve read upping that to 51200 will sometimes yield speedier results:

sudo dd bs=51200 if=/dev/disk3s3 of=/Volumes/Work/LastDitch-DD-02.dmg conv=noerror,sync

In my case it did not, primarily, I believe, because there was a problem in the beginning of the drive, and dd was having trouble moving past that spot. So another thing you can tell dd to do is to skip a certain portion of the drive, say the first 2 GBs:

sudo dd bs=51200 if=/dev/disk3s3 skip=2000000 of=/Volumes/Work/LastDitch-DD-03.dmg conv=noerror,sync

Finally, you can also tell dd to only write in 1 GB chunks, using the count option:

sudo dd bs=51200 if=/dev/disk3s3 count=1000000 skip=2000000 of=/Volumes/Work/LastDitch-DD-03.dmg conv=noerror,sync

I was getting some good results after having skipped the first 2 GBs — apparently they were really damaged — so I decided to write a script that would skip the first 2 GBs and then begin writing out 1 GB chunks of data. It would’ve looked something like this:

sudo dd bs=51200 if=/dev/disk3s3 count=1000000 skip=2000000 of=/Volumes/Work/LastDitch-DD-Chunck-01.dmg conv=noerror,sync
sudo dd bs=51200 if=/dev/disk3s3 count=1000000 skip=3000000 of=/Volumes/Work/LastDitch-DD-Chunck-02.dmg conv=noerror,sync
sudo dd bs=51200 if=/dev/disk3s3 count=1000000 skip=4000000 of=/Volumes/Work/LastDitch-DD-Chunck-03.dmg conv=noerror,sync
...

Etc, etc, up to the 40 GBs needed to scour the drive. I never got to write the script, though, because the last dd command seized up and the drive began making the clicking, knocking and whirring sounds of its agonized and tortured death. It was quickly dying. We could do no more.

At this point, mainly for my own edification, I decided to see what could be done outside the confines of my home office. I decided to get a quote from Drive Savers.

Hardware-Level Attempt
Drive Savers, perhaps wisely, does not list prices for their services on their website. To get an estimate you have to give them a call. When I called them I was greeted by a very friendly and helpful service person — yes, person — which was really nice. The last thing you want to deal with when you’re having a mechanical failure is a machine. The person on the other end of the line asked me a few basic questions to gauge what state the drive was currently in, things like what attempts I had made to rescue the data, would the drive mount, and the like. After entering this info into her systems she directed me the “Tips, Techniques and Solutions” page on their website (very useful — love the drive sound audio samples), stressing above all that in order to have the best chance of recovery at this point the drive should not be powered on again. She also offered up some information about the company and what they do: For one, they started with Mac data recovery and are an all-Mac shop, which surprised me a little. She also pointed me to information on the Drive Savers clean room, a vital part of data recovery at the hardware level. She then took my email and contact info and gave me both a written and verbal estimate of how much I could expect to spend should I decide to go ahead and have Drive Savers attempt to save my data (I don’t think they’ll actually save the drive). All in all it was a very pleasant and informative experience. Normally I am loathe to use the phone for business, but Drive Savers really seems to know what they’re doing, at least when it comes to pre-sales customer service, and that counts for a lot in my book.

This is, of course, all prep for the fact that, if you do want to make the attempt at data recovery, you’ll be expected to drop a significant amount of money. This is hardly surprising. Those clean rooms don’t look particularly cheap to build or maintain. And if data recovery at the hardware level is anything like it is at the software level, it is a laborious and time consuming process. I was given a range of prices ($500-$2700 dollars) and told that the cheapest I could expect to get away with — the economy plan, which isn’t as fast as some of the other, more expensive plans — was $500 dollars. But it was likely I’d pay somewhere closer to the upper third of the range, more like $1500 to $2000 dollars. It all depended, of course, on how much data Drive Savers could recover.

I didn’t really find these prices particularly surprising. I’d long heard how much such a recovery could cost. That it would be pricey. I was glad that I was not in a situation that required me to fork out this amount of money. I’m glad such a service exists for the odd catastrophe, though I hope never to have to use it. Drive Savers’ website offers advice on keeping backups:

“Backup strategies:
* Invest in redundant backup systems
* Establish a structured backup procedure to make copies of all critical data files, using software compatible with the operating system and applications
* Periodically test the backups to verify that data, especially databases and other critical files, are being backed up properly
* Keep at least one verified copy of critical data offsite”

Sage advice, all. Take it from those who know all too well.

The Belly of the Beast
Once we’d decided not to use a hardware data recovery service the only thing left to do was spec out, buy and install a new hard drive. This wasn’t terribly difficult, but as is so often the case, there was the odd snag or two.

Before we even bought a drive, I wanted to see how hard it would be to open the PowerBook for servicing. If it was going to be a bear — and some PowerBooks are certainly easier to crack than others — I’d let the fine technicians at Tekserve do the job. So I went in search of manuals and instructions for this particular model of PowerBook. Without too much trouble I was able to locate, at Apple’s site, the manual for our 1.67 MHz, 15″ Aluminum PowerBook. It contained no instructions for hard drive replacement, which is generally a sign that Apple would rather you not attempt the repair yourself. That got me a little worried.

Finally, however, I found instructions — great instructions, no less — at the venerable — awesome, actually iFixit.com. iFixit, for those of you who don’t know, provides step-by-step, illustrated guides on taking apart and performing repairs on Apple hardware. For free. They’re amazing. I feel guilty not buying anything from their site. Oh yeah, they also sell parts, tools and service as well. I love them. And from what I could see, the repair would be tedious — lots of screws — and would require a trip to the hardware store — blasted tiny hex screws! — but it would be doable. Still, taking things one step at a time, I thought I’d perform the teardown before buying the drive. Just in case.

And perform I did. Using iFixit’s excellent guide, I was able to crack the PowerBook in short order. I was ready to buy a drive.

Buying a Drive
There are two things SysAdmins typically are, particularly when it comes to technology: cheap and lazy. Hunting for a replacement drive brought both of these qualities in my personality to bear. I was looking for the cheapest replacement I could find, at the location closest to my house, a SysAdmin’s dream hunt. The closest proper computer tech shop to me is Tekserve, with Best Buy a close second. Tekserve doesn’t list what bare drives they carry, if any. But Best Buy seems to have the goods. But Best Buy is still a good half hour train ride, so I did some physical recon at my nearest Radio Shack, which happens to be right around the corner. They informed me that, though they did not have any bare drives in stock, they did have portable USB drive on sale. Drives from which I could pull and the internal component and install it in the now drive-less PowerBook. In fact, they had a 160 GB Iomega Prestige for less than a bare drive would have run me at Best Buy — a mere $75 clams post-sales-tax. Not bad. I took it.

I’d like to pause here and see if anyone can guess why this didn’t work out for me. You have pretty much all the data you need in this article to figure it out. But don’t feel bad if you can’t. The good lord knows I surely didn’t. I’ll wait a minute… Pretend there’s Jeopardy countdown music playing… Aaand…

Okay. Did you guess it?

I got the drive home, popped it out of its case and went to put it in the open PowerBook. But it didn’t fit. (Have you guessed it yet?) Here’s the thing: PowerBooks use 2.5″ ATA drives (Parallel ATA, or PATA), but drives in today’s externals are all now SATA (Serial ATA) drives. Blast!

Oh well. At least it was cheap.

Another quick look at the web revealed that all the bare drives at Best Buy were SATA as well. Blast again!

The nearest ATA drive I could find was at J&R, which is all the way downtown, almost at the very tippy-tip of Manhattan — far. So that’s where we went.

Once we got back, we installed the drive and — the very first thing to go right all day — it worked. Perfectly. Things were finally looking up.

Once we had installed the drive it was simply a matter of formatting it, installing the latest version of Leopard (which is all we ever wanted to do in the first place) and copying over the rescued and reconstructed data. Oh, did I mention that the reason the client wanted Leopard was for Time Machine? Yup. Backups. Great timing. So we set up Time Machine as well. All that went exceedingly smoothly and our repair is, at last, complete. Whew! What an ordeal!

But, man, did I ever learn a lot.

The Life and Death of Hard Drives
So yes, drives die. How they die, though, is almost as important as how they lived, and certainly as interesting. It’s somewhat comforting to know that this drive, while quite dead indeed, did not die in vain. Rarely have I had the opportunity to learn so much about practical drive recovery. I have that PowerBook drive — specifically its death, in fact — to thank for my lesson.

Most recently, something seems to have taken a large (by which I mean everything) bite out of a very important CSS file. See, we tend to use Coda to build sites at our house, and we tend to work over the network as the most expedient means to that end. Now, working on a website over the network is not without its perils, as I’m sure you’re aware. Particularly if you’re working wirelessly, and particularly if you’re working on a server of unknown reliability. So, a very awesome someone I know (okay, yes! I have a girlfriend!) was doing exactly that when all of a sudden her CSS file appeared to be completely empty. Mind you, she was not working on the file. She merely had it open while she worked on another document in another tab. But after switching to the CSS tab, the CSS file — which she’d been working on obsessively for about a week — appeared to be empty.

Now I’ve had the same thing happen to me after a network dropout — or, more likely, a server disconnect — and the solution in my case was to simply shut down and restart Coda. Mine was largely a cosmetic issue brought on, I assume, by Coda’s inability to reconnect to the documents after a disconnect. So I told her to simply restart Coda, confident that the problem would correct itself. But it didn’t. Even after restarting Coda, still no CSS joy. The file was there, but it was completely empty!

This is the point at which panic generally sets in. (And no, that is not a reference to the makers of Coda.)

Panic

If there’s anything I’ve learned in my near-ten years of professional systems work, it’s that data is rarely ever completely wiped out in a single stroke. And if there’s anything else I’ve learned, it’s not to panic. So I coolly, calmly set about the task of recovering the file while my exhausted and infuriated sweetheart went to bed.

The first thing I did was to check the server to see if any backups had been made. I know that her provider, and some of the software she uses, make automatic backups from time to time. So I downloaded anything and everything I could find from the server that might prove useful, including a backup of the entire site for safekeeping. I soon discovered that there was nothing even remotely recent enough to contain the missing CSS file. So I started looking in the local home account, first by grepping for anything with “css” in the name. Some Coda cache files came up, some of which were fairly recent, but none failed to yield the data I was searching for. I searched /tmp as well, to no avail.

Finally, after a couple of hours of downloading and grepping and searching and hoping, I was about ready to give up. As a last ditch effort I decided to use the find command on the entire local hard drive:
find / -name *.css*

This command will search the entire file system for any file whose name contains the string “.css.” And it turned out to be the winner. The command yielded a ton of useless results, many of which came from application documentation. But in the end a Coda cache file turned up in:
/private/var/tmp/501

Of all places!

Moreover, this file had a time stamp very near the time of the disappearing data. So I made a copy of it (okay, I made, like, four copies of it) and uploaded it to the server. The next day my sweetie confirmed: I’d found the file! The day was saved!

So remember, people: Stay calm, and always try find before giving up the ghost. And for poops sake, make a backup!

Whew! That was close!

This no longer works. Now we must restart autofs. To restart autofs on Mac, do this:

To be additionally thorough, though this should not be necessary, you could also restart automount, which now looks slightly different (note the “d”, which is new):

None of this is surprising, but then again, if you’re not sure you’re doing it right (like you run the command and nothing happens and you want to be sure you’re doing the right thing) it helps to have it written down somewhere.

Enjoy!

I was attempting to set a Linux system to authenticate via a freshly-built LDAP server — something I’ve done many, many times — and it just wasn’t working. I could authenticate and log in fine via the shell, but no matter what I tried, whenever I would attempt to log in to Gnome, I’d get an error message saying that my session was ended after less than 10 seconds, that maybe my home account was wonky or I was out of disk space, and that I could read some error messages about the problem in a log called .xsession-errors in my home account.

Of course, certain that my home account was fine and that I had plenty of disk space, the first thing I checked was the .xsession-errors log, which yielded little useful information, and which information led me on a complete and utter wild goose chase. From everything I could glean from this rather sparse log, there seemed to be a problem with Gnome or X11 not recognizing the user. I showed the error to some UNIX-savvy co-workers, one of whom demonstrated that, when booting into run-level 3, logging in and then starting X, login worked fine, thus proving my hypothesis. So began several days of research into Linux run-levels, Gnome, X11, PAM, NSS Switch and LDAP authentication on Linux. All of which was exceptionally informative, but which, of course, failed to yield a positive result.

The final, desperate measure was to scour every forum I could, and try every possible fix therein. And, lo and behold, there, at the bottom of some obscure post on some unknown Linux forum (okay, maybe not that unknown), was my answer: set the default shell. Could it be so simple?

But wait, wasn’t the default shell set on my server already?

I checked my server, and sure enough, because of a typo in my Record Descriptor header, the default shell had not been set for my users. Seems X11/Gnome needs this to be explicitly specified in an LDAP environment, because in said environment it is (for some reason that remains beyond me) unable to read the system default.

Setting the default shell for users on my LDAP server (yes, it is a Mac OS X Server) did the trick, and I can now log in normally to Linux over LDAP.

So, after days of researching a problem the solution all boiled down to one, dumb, overlooked setting on my server, a fact I found referenced only at the bottom of some strange and obscure internet forum. Sound familiar? What, pray tell then, should we call this phenomenon? We really need a term for it. Or a perhaps an axiom? Maybe a law or a razor or a constant. Something like:

“For every seemingly complex OS problem there is almost always an astoundingly simple solution which can usually be found at the bottom of one of the more obscure internet forums.”

A corollary of which might go something like:

“Always check the bottoms of forums first.”

We’ll call it Systems Boy’s Razor. Yeah, that should do nicely.

If anyone has any better suggestions here, I’m always open. Feel free to let ‘em rip in the comments. Otherwise, check your default shells, people. Or at least make sure you have them set.

I’m also getting fairly adept at making packages. A good many of my packages are just scripts that make settings to the system, so I’m getting pretty handy with the bash and quite intimate with dscl. But, perhaps most importantly, I’m learning how to make all sorts of settings in Leopard via the command-line that I never knew how to do.

The toughest one so far has been file sharing. In our lab we share all our Work partitions to the entire internal network over AFP and SMB. In the past we used SharePoints to modify the NetInfo database to do so, but this functionality has all been moved over to Directory Services. To complicate matters, SAMBA no longer relies simply on standard SMB configuration files in standard locations, and the starting and stopping of the SMB daemon is handled completely by launchd. So figuring this all out has been a headache. But I think I’ve got it!

Setting Up AFP
Our first step in this process is setting up the share point for AFP (AppleFileshareProtocol) sharing. This wasn’t terribly difficult to figure out, especially now that I’ve been using Directory Services to create new users. To create an AFP share in Leopard, you use dscl. Once you grok the syntax of dscl it’s fairly easy to use. It basically goes like this:

command node -action Data/Source value

The “Data Source” is the thing you’re actually operating on. I like to think of it as a plist entry in the database — like a hierarchically structured file — which it basically is, or sometimes I envision the old-style NetInfo structures. To get the needed values for my new share, I used dscl to look at a test share I’d created in the Sharing Preferences:

dscl . -read SharePoints/TEST

The output looked like this:

dsAttrTypeNative:afp_guestaccess: 1
dsAttrTypeNative:afp_name: TEST
dsAttrTypeNative:afp_shared: 1
dsAttrTypeNative:directory_path: /Volumes/TEST
dsAttrTypeNative:ftp_name: TEST
dsAttrTypeNative:sharepoint_group_id: XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXX
dsAttrTypeNative:smb_createmask: 644
dsAttrTypeNative:smb_directorymask: 755
dsAttrTypeNative:smb_guestaccess: 1
dsAttrTypeNative:smb_name: TEST
dsAttrTypeNative:smb_shared: 1
AppleMetaNodeLocation: /Local/Default
RecordName: TEST
RecordType: dsRecTypeStandard:SharePoints

Okay. So I needed to use dscl to create a record in the SharePoints data source with all these values. Fortunately, the “sharepoint_group_id” is not required for the share to work, because I’m not yet sure how to generate that number. But create the share with all the other values and you should be okay:

sudo dscl . -create SharePoints/my-share
sudo dscl . -create SharePoints/my-share afp_guestaccess 1
sudo dscl . -create SharePoints/my-share afp_name My-Share
sudo dscl . -create SharePoints/my-share afp_shared 1
sudo dscl . -create SharePoints/my-share directory_path /Volumes/HardDrive
sudo dscl . -create SharePoints/my-share ftp_name my-share
sudo dscl . -create SharePoints/my-share smb_createmask 644
sudo dscl . -create SharePoints/my-share smb_directorymask 755
sudo dscl . -create SharePoints/my-share smb_guestaccess 1
sudo dscl . -create SharePoints/my-share smb_name my-share
sudo dscl . -create SharePoints/my-share smb_shared 1

This series of commands will create a share called “My-Share” out of the drive called “HardDrive.”

After modifying the Directory Services database, it’s always smart to restart it:

sudo killall DirectoryService

And we need to make sure AFP is running by starting the daemon and reloading the associated Launch Daemons:

sudo AppleFileServer
sudo launchctl unload /System/Library/LaunchDaemons/com.apple.AppleFileServer.plist
sudo launchctl load -F /System/Library/LaunchDaemons/com.apple.AppleFileServer.plist

Not the easiest process, but not too bad. SMB was much tougher to figure out.

Setting Up SMB
Setting up SMB works similarly, but everything is in a completely different and not-necessarily standard place. To wit, Leopard has two different smb.conf files: one that’s auto-generated (and which you should not touch) in /var/db, and one in the standard /etc location. Fortunately, it turned out, I didn’t have to modify either of these. But still, it led to some confusion. The way SMB is managed in Leopard is rather roundabout and interdependent. Information about SMB share is stored in flat files — one per share — in /var/samba/shares. So, to create our “my-share” share, we need a file named for the share (but all lower-case):

sudo touch /var/samba/shares/my-share

And in that file we need some basic SMB info to describe the share:

Next — and this was the tough part to figure out — we need to modify one, single, very important preference file that basically informs Launch Services that SMB should now be running:

This command modifies the file com.apple.smb.server.plist in our /Library/Preferences/SystemConfiguration folder. That file is watched by launchd such that when it is modified thusly, launchd knows to start and run the smbd daemon in the appropriate fashion. Still, for good measure, I like to reload the LaunchDaemon for the SMB server by hand. Don’t need to, but it’s a nice idea:

sudo launchctl unload /System/Library/LaunchDaemons/com.apple.smb.server.preferences.plist
sudo launchctl load -F /System/Library/LaunchDaemons/com.apple.smb.server.preferences.plist

That’s pretty much it! There are a few oddities: For one, the new share will not initially appear in the Sharing Preferences pane, nor will the Finder show it as a Shared Folder when you open the window.

But the share will be active, and all will be right with the world after a simple reboot. (Isn’t it always!) Also, if you haven’t done it already, you may have to set permissions on your share using chmod in order for anyone to see it.

I was kind of surprised at how hard it was to set up file sharing via the command-line. But I’m glad I stuck with it and figured it out. It’s good knowledge to have.

Hopefully someone else will find it useful as well.

/System/Library/CoreServices/RemoteManagement/ARDAgent.app/Contents/Support

/usr/sbin

Really now. Was that so hard?

sudo killall -HUP ARDAgent

Ironically, you can also send the command to the offending system via ARD itself. Just be sure you remove the sudo and send it as root.

Restarting ARD Via ARD
(click image for larger view)

That’s it! Just another helpful tip from your friendly neighborhood Systems Boy.

Please resume your normal activities.

But moving and planning the physical aspects of the new lab has only been a portion of what I’ve been working on. This renovation has been the perfect opportunity to rebuild our network infrastructure, and part of said rebuilding has resulted in the near completion of our authentication unification project. At this point we’ve gone from eight different authentication servers — that is, anytime we created a new user, we had to do so on eight different systems — all the way on down to two. Which means that now, anytime we create a new user, we do so on two machines.

Our goal is to get it down to one, hopefully before the Fall semester begins. Our mail server is proving to be the most difficult machine to get working with LDAP authentication, mainly because it authenticates mail users through the wonders of some weird combination of authd, Courier and PAM, and we’ve yet to crack the magical code that gets these all working in tandem via LDAP. Aside from Mail, though, everything is done. So I thought I’d take a bit of my hard-earned vacation and loosely describe to you how it’s all working.

Before I start I’d like to just acknowledge all the help I’ve had from my fellow SysAdmins in the department. I had a huge amount of assistance on the *NIX server side of things, as well as with network infrastructure and even some last-minute PHP finagling without which this project would have taken significantly longer. In fact, all I really had to do was build the authentication servers and clearly articulate what I wanted. I’m extremely grateful to everyone who helped out.

The little bit of network infrastructure I mentioned is our DMZ. We now have a proper — and more importantly, properly secured — DMZ on which to place an authentication server. I won’t go into too much detail here, but suffice to say, having a secure DMZ gives us all kinds of options for authentication between internal and external networks, and makes me feel a whole lot better about using Mac OS X Server as our authentication system for both networks.

Yes, we are using Mac OS X Server to authenticate our entire network. The reason is because Mac OS X Server is the most mature and usable implementation of LDAP for user authentication available on the market today. Is it perfect? No. Is it completely secure? Probably not. Is there anything that even comes remotely close to being able to handle the complexities of user management and database redundancy across platforms with such remarkable ease-of-use? Nope. Nothing. We tried building our own custom LDAP server, which would have been excruciating, and would have taken forever. We tried Red Hat’s Directory Server, which looks like it will eventually turn into something to match Mac OS X Server, but which just wasn’t yet up to snuff. Nothing matched Mac OS X Server, which did everything we wanted it to, right out of the box and with a minimum of fuss. In fact, once the user database is built, building a Mac OS X master or replica authentication server is a complete and total breeze. At the time of our building and testing it was really the only practical option.

So, here in a nutshell, is what we have:

Internal Network
All authentication originates from the internal network. Passwords can only be changed from the internal network at this time, which is by design. Systems on the internal network include:

• Master Authentication Server
  Hosts authentication for… Well… Everything, really. This is essentially the same server we used all last year for all our internal authentication needs for Mac, Linux and Windows workstations. It’s now being used to push authentication to the external network as well.
• Internal Replica Authentication Server
  This provides replication of the Master. Should the master fail, the Replica is intended to pick up services (though this doesn’t always work perfectly).
• File Servers
  We have two file servers on the internal network — a Mac and a Linux box — both of which authenticate directly against the Master.
• Workstations
  We have about 30 Mac, Windows and Linux machines all authenticating to the Master.

DMZ
The DMZ sits between the Big Bad Internet (BBI) and the internal network. It has its own firewall that is fairly strict about what can get in from the BBI. All DMZ authentication originates from the internal network, but is provided by a single server which sits on the DMZ. Systems on the DMZ include:

• External Authentication Server
  This server is also a replica of our Master, but it’s not intended as a failsafe. Rather, it provides authentication services to the entire DMZ. It gets its user database, of course, from the Master. But for other systems to bind to an LDAP server, its role must either be “Master” or “Replica.” Setting the role to “Connected to a Directory Server” won’t work. In addition to sitting on our DMZ, which is properly firewalled against the harsh realities of the Big Bad Internet (BBI), this system also makes use of its own strict local firewall for an extra added layer of security. Also, all replication communication between Replica (DMZ) and Master (Internal) is encrypted.
• Data Server
  In addition to unifying authentication, we’ve also consolidated data storage and access wherever possible. In the past, for instance, movies streamed from the Quicktime Server were stored on that machine’s local drive. Web sites were stored on our web server. So, building a web site that used Quicktime Streaming required users to log into two separate machines — the Web Server and the Quicktime Streaming Server. Now we’re storing all user-generated content on a separate, dedicated machine — our Data Server — and sharing that machine out to the various servers via NFS. Centralizing this data store means users have only to log on to one server for anything they ever want to do. And also that only that server needs to authenticate users. And yes, that server authenticates them via LDAP on our External Authentication Server. All neat and tidy. Internal and external home account data is still segregated, however — users still have separate internal and external data storage. Though, if we could figure out how to do it securely, this could change.
• Quicktime Streaming Server
  This machine also uses its own local firewall. It gets its user database from our External Authentication Server over secure channels using the “Connected to a Directory System” as its role currently. Ultimately, however, because of the Data Server, this machine will not need to authenticate users. We are leaving the ability open temporarily to accommodate legacy users.
• Drupal CMS
  Our new Community site is built on the Drupal engine. We’re using the LDAP module to authenticate to the External Authentication Server. Drupal’s LDAP module is simple and easy to set up, as is the Drupal system as a whole. So far we’re very happy with it.
• Computer Reservations System
  This is a custom web app built long ago by a former student. We’ve (and by “we” I mean my colleague) basically hacked the PHP code to authenticate
  via LDAP rather than MySQL.
• Mail Server
  Currently not authenticating to the External Authentication Server. We’re working on this and hope to have it working by the beginning of the school year.

The Future
Yes, there’s more we want to do. It’s always amazing how, once you’ve completed something, you immediately start seeing ways to make it better.

• More Redundancy
  Ultimately, in addition to the Replica, I would also like to automate a clone of the Master’s boot drive to an external firewire drive as sort of an ultimate safety. Should anything ever go wrong with the Master, I simply plug the firewire clone into virtually any Mac system on the internal network and I’m back on my feet. It might also be wise to have some sort of failsafe for external authentication as well.
• More Security
  while our setup is fairly secure right now, there are a few areas I’d like to beef up even more when I get a chance. In particular, our CMS connection is not as secure as I’d like it to be. And ultimately I’d like to harden every machine on the DMZ to the best of my ability.
• More Unification
  Anything else we can unify — and at this point that’s mostly internal and external data — I’m open to considering. It’s going to be really interesting for me to look critically at what we’ve done so far and find the flaws and refine the system. But I’ll constantly be looking at ways to simplify our current setup even further without compromising security. The easier our network is to use, the more useful it becomes. We’ve come a long way, but I’m sure we can find even better ways to do things.
• More Services
  Now that we have an infrastructure in place for user creation, we can add services freely to our network without the worry of creating users for said services. New services need only the ability to authenticate via LDAP. We’re already planning an equipment checkout system, and possibly some calendaring systems.

So, I’ve just finalized the master authentication server. It’s done. Built. Finished. Kaput. The rest of our servers are still in various states of finality, and we have until September to lock them down. But right now, unified authentication is, for all intents and purposes (and with the exception of mail), working. And we couldn’t be happier. The ultimate test will be, of course, letting users loose on this new infrastructure. I’m betting they’ll like it almost as much as we do. At least the ones who know the old system. New users will be none the wiser. Ain’t that always the way?

*Sigh*

And by golly, that’s just what I did. Here are a few Automator workflows that do, more or less what the afore-linked methods do. To me, the advantage of the Automator approach is that you don’t need to install anything. It’s all baked in. Which means you don’t ever need to update anything either. Nice. Simple. And, yeah, kind of the whole point of Automator.

So here you go. Maybe someone will find this useful, if for nothing other than as an exercise in creating contextual menu functionality with Automator. Or skinning a cat multiple ways. Or something. To use this, download the .zip file, unzip it and place it in:
~/Library/Workflows/Applications/Finder

NewTextFile Workflow

It should become active immediately.

Also, here are a couple variants. One will create a text file, and then open it in TextWrangler (if you have TextWrangler, and if you don’t, go get it now); the other creates a Word document, and opens it in Word. I’m far to lazy to completely duplicate the functionality of NuFile. But if you examine these workflows, you can at least see now how that would be possible (in fact, fairly easy) to accomplish.

NewTextFile Workflow Variants

I actually think it would be great if Apple made it drop dead simple to create true contextual menus for the Finder. Fortunately, Automator gets us pretty close.

Oh, yeah, and since this is technically script writing, and since I haven’t posted to that series in some time, we’re gonna go ahead and call this a Script Sharing post. Deal with it.

Right. Good night.

UPDATE: Revised March 31, 2007, 3:00 PM
Stephan Cleaves has added yet another implementation of this idea. He’s using a combination of Automator and AppleScript. I certainly think his implementation is better than mine in a few ways. Certainly more full-featured. It will prompt for a file name, for instance, and takes pains not to overwrite a preexisting file with the same name. Nice. But we’re taking very different approaches to the same idea (his version places a file in the front-most Finder window, my version places it in the right-clicked folder), and he was confused by my approach. After speaking to him via comments on his blog, I realized that some clarification as to how my workflow is actually constructed might be in order.

Basically, my workflow takes the folder selected in the Finder as input and assigns that input to the variable “$@”. That variable and the for loop in my workflow are automatically generated by Automator when you select “as arguments” from the “Pass input:” field in the “Do Shell Script” action. It’s how you get the context (the selected folder) passed to the script. Apparently Automator takes “$@” as the variable for “the folder you just selected” whenever there’s no input from a previous action. This was something I learned while fiddling around with all of this, and it’s really my favorite part. The coolest thing for me here, really, was figuring out how to pass the context — i.e. the right-clicked folder — to an Automator “Do Shell Script” action. This opens up worlds of potential.

Finally, as I said, the for loop in the action is auto-generated by Automator. The workflow will work almost as well with the simple script:
touch “$@/NewText.txt”

Using the for loop, however, allows you to create a new text file in multiple folders by selecting said folders and running the workflow.

It’s really kind of amazing how many ways there are to do this. Wow. Fun stuff.

/var/run/openldap-slurp/replica

and doing so did reveal errors like:

ERROR: Type or value exists: modify/add: memberUid: value #0 already exists

The solution was to again demote the replica to standalone status and then archive all the files in:

/var/run/openldap-slurp/replica

to anywhere else. I put them in a folder called “old.” Just get ‘em out of the way. Once this was done I was able to promote my replica without receiving error messages.

Yay! That wasn’t too bad.

Oh, and you may be asking yourself how I knew to do this. Well, to be honest, I don’t really remember. I just know that at some point in the past there was a problem I’d had with a replica and it was caused by stale files. So, since my ultimate goal was to start from scratch, I just got everything out of the way. And lo and behold. It worked. Sorry for the voodoo explanation, though. I wish I could be more explicit. Hell, I wish I fully understood what I was dealing with. But I don’t. And, though it pains me to say this, I don’t have time to figure it out.

But y’know? I’ll take the cure even if I don’t know what caused the disease.