Now, to remind, we don’t really use AFP networked home accounts for most users. Our users’ home accounts live on an NFS server — a separate machine from our authentication server — which is auto-mounted on each client at boot. The only value the authentication server stores for most users’ home accounts is where those home accounts can be found on the client machines, which in our case is /home. So I haven’t had to worry too much about AFP network home accounts. Until last week.

There is one exception to the above standard. Certain Macromedia products do not function properly when the user’s home account is located on our NFS server, for some reason. In particular, Flash is unable to read the publishing templates, effectively disabling HTML publishing from the app. This has been a long term problem and has affected every version of Flash since we moved to our NFS home account server almost three years ago. Our solution has been to create a special user — the FlashUser — whose home account lives in an AFP network home account. When people need to work with Flash, they are encouraged to use this FlashUser account so that they can use the publishing features. This is inconvenient, but it works and we’re used to it, so we keep doing it until we find a better solution. Unfortunately, when I built my master authentication server (actually, when I rebuilt it) I forgot to add the FlashUser. The teacher of the Flash class eventually came to my office and asked about the account, and I told him it should just take a minute or two to get it set up. Boy was I wrong.

The FlashUser account was always a simple AFP network user account. The user’s account information and actual home account data were both stored on the same server, the data shared via an AFP share point, and configured in Workgroup Manager’s “Sharing” pane as an auto-mounted home folder. According to this article guest access must be enabled on the AFP share to auto-mount. Well, that’s new in 10.4, and I had no idea. And beyond that, I think it sucks. Why should guest access be enabled on a home account share point? This didn’t used to be the case, and it seems way less secure to me in that by doing this you open the entire AFP home account share to unauthenticated access. Bad. Why in the hell would they change this? Not only is it less secure, but it breaks with tradition in ways that could (and in my cases did) cause serious headaches for admins setting up AFP networks home accounts.

Fortunately, after many hours of trial and error, I discovered a way to accomplish the same thing without enabling guest access on the share. It is possible, but it’s quite a pain. Nevertheless, it’s what I did.

Guest access can be foregone if trusted directory binding is set up between the client and the computer (which still makes no sense to me. You either have to go totally insecure, or set up an insanely secure system. Seems like we could skip the trusted binding thing if you’d just let us set up the shares sans guest access like we used to do.) Trusted binding is a bit of a pain to set up in that, as far as I know at this point, the only way to set it up is to go to every machine, log in, and run the Directory Access application. Apple really, really, really needs to give us admins some command-line tools for controlling DA. The current set is paltry at best, though I do need to look and see if there is one for setting up client-server binding (there might be, in fact it might be called dsconfigladp though this tool can not be used for setting authentication sources, for some ungodly reason, and I have yet to try it for the purposes of directory binding). But before you set up your clients, you must be sure “Enable Directory Binding” on your server is checked in the Server Admin application. By default it’s not. And, at least in my case, after enabling directory binding, I had to restart my server. Fun stuff. Also, I’m fairly certain this requires a properly functioning Kerberos setup on the server, so if you’re just starting out, be sure to get Kerberos running right.

Directory Access: Enable Directory Binding
(click image for larger view)

Next you need to go to your client machines one by one and bind them to the server. If you’ve already joined your clients to a server for authentication, you can simply navigate to the server configuration in DA, click “Edit…” to edit it, and you’ll be presented with a “Bind…” button that will walk you through the process. If you haven’t yet joined a client you will be asked to set up trusted directory binding when you do. From there you need to enter the hostname of the client machine, the name of a directory administrator on your server, and that user’s password. In my case, I needed to also reboot the client machine. Like I said, kind of a pain.

Directory Access: Configure LDAP
(click image for larger view)

Directory Access: Edit Server Config
(click image for larger view)

Directory Access: Hit “Bind…” to Set Up Trusted Directory Binding
(click image for larger view)

But that’s it. You’re done. (Well, 25 computers later, that is.) I now have my F

lashUser set up again, and auto-mounting its home account in the usual way (none of that “Guest Access” shit), albeit with considerably more effort than it took in Panther. This is, in my opinion, just one more reason to hate Tiger, which I generally do, as long-term readers are well aware. It’s another case in which Tiger has gotten harder and more complicated to use and set up, with no apparent advantage, or at least no immediate one.

I can only hope this is going somewhere good, because from my perspective the basic things I’ve come to rely on in both Mac OS X Server and Client (can you say “search by name?”) have gotten harder to do in Tiger. Significantly harder. And that’s too bad, ’cause that’s just not what I’m looking for from Apple products. If I wanted something difficult to use, I’d switch to Linux. (I’d never switch to Windows.) And if Apple software continues along its current trajectory — the Tiger trend towards a more difficult OS — and Linux software continues along its current trend towards easier software, you may see more Linux switchers than ever. Apple’s draw is ease-of-use. The more they move away from that idea in the implementation of their software, the less appealing they become, to me personally, as a computing platform, and the less distinguishable they become from their competitors.

But for now, I’m sticking by my favorite computer platform. Panther was an amazing OS — it truly “just worked”. It’s kind of sad that Tiger has been the OS that’s made start considering, however peripherally, other options. Here’s hoping they do better with Leopard.

BTW, here are a couple more links of interest regarding the topic of AFP newtork home accounts in Tiger. I don’t have time to read them right now, but they may prove interesting at some point.

Creating a Home Directory for a Local User at a Server
Creating a Custom Home Directory
Automatically Mounting Share Points for Clients
Setting Up an Automountable AFP Share Point for Home Directories

UPDATE:
A fellow SysAdmin and blogger, Nigel, from mind the explanatory gap (one of my systems faves) has some beefs with this article, and shares some of his own experiences which are quite contrary to what I’ve reported. Just to be clear, this article reflects my own experiences, and there’s a bit more to my own scenario than I shared in the article, mainly because I didn’t think the details were important. They may be, and I discuss it at greater length in my response to Nigel’s comment. Thanks, Nigel, for keeping me on my toes on this one.

I’m not really sure what’s going on here, but if I get to the bottom of it, I’ll certainly report about it, either here or in a follow up post. But please read the comments for the full story, as there’s really a lot missing in the article, and things start to get clearer in the comments.

On the plus side, I was able to observe some good behavior for a change on my Mac clients. In the last article I’d mentioned that it’s the clients that are responsible for keeping track of the master and replica servers, and that they get this info from the master when they bind to it, and that this info was probably refreshed automatically from time to. Well, this does indeed seem to be the case. Mac clients do periodically pull new replica info from the master, as evidenced by the presence of the replica in the DSLDAPv3PlugInConfig.plist file where once none existed, and on machines I’d not rebound. Nice. Guess I won’t be needing to rebind the Mac clients after all. For those interested in theories, I believe this gets taken care of by lookupd. If I understand things correctly, lookupd manages directory services in Mac OS X, particularly the caches for those services. Mac OS X caches everything, and in Mac OS X, even Directory Service settings are cached. DNS. NetInfo. SMB, BSD, NIS. All cached. Most of these caches — like DNS, for example — have pretty short life spans. But some services don’t need refreshing so often. Things like Open Directory services stay in cache for a bit longer. There’s even a way to check and set the time-to-live for various service caches, but I’m not quite there yet. But I believe it’s lookupd that grabbed the new settings from the server, or at least expired the cache that tells the client to go get those settings. In any case, there’s a lookupd command I’ve found increasingly handy if you’ve just made changes to certain settings and they’re not yet active on your system:

sudo lookupd -flushcache

This will, obviously, flush the lookupd cache, and refresh certain settings. For instance, DNS changes sometimes take a bit of time to become active. This command will take care of that lag. My favorite use, though, is on my server. See, when I create a new OD user, I use the command dscl. Apparently, using the Workgroup Manger GUI will flush the cache, and the user will be instantly recognized by the system. Smart. But if, like me, you use a script that calls dscl to add or remove an OD user (remember, OD users are managed by a directory service, as are local NetInfo users, for that matter), the system won’t become aware of said user until the cache is flushed. I used to add a new user, id them, and sometimes get nothing for the first few minutes. Or freakier, delete a user and still be able to id them in the shell. Until I found out more about lookupd I thought I was going crazy. Now I just know to include the above command in my AddUser and DeleteUser scripts. Nice. Nicer still to know I’m not losing my mind. At least not in the case of adding or removing users.

Anyway, when I get ’round to my final Windows services tests, I will post an update.

God, I’m sick of this replica shit.

Getting this working has been a saga of nearly epic proportions. I’d long suspected that my replica problems were due to latent Kerberos problems on the master, which occurred in the initial stages of its installation and setup, and I still believe this is indeed the case. Loathe as I was to rebuild a seemingly perfectly working master, I was considering trying it when circumstances beyond my control, and I believe completely unrelated to the problems I was having, forced my hand. Towards the end of the week the master server committed suicide.

I was doing some work on the server when I noticed that the system drive was inexplicably full. Odd, considering that this is a 20GB partition, and the system, apps, and my files should have been using no more than 6GB. Where had all that space gone? Some quick investigating led me to the folder /var/spool/atprintd which was hogging about 10-15GB of disk space. (I’m recalling all this from memory, so details might not be perfectly accurate.) A surely related symptom of all of this was that, whenever I went to set print preference for computers in Workgroup Manager’s “Computers” list, I’d get a spinning wheel, but never a graphic or access to the control. Didn’t think much of ’til I discovered this print queue spooling dangerously out of control. Still, no big, I figured, and I deleted the spool files, and deleted the queue (I wasn’t even using it) and rebooted the machine. Problem solved, right? Wrong.

After reboot the master server formerly know as “Open Directory Master” listed as it’s role “Connected to a Directory System” in Server Admin’s Open Directory settings pane. Huh? Exactly what Open Directory System was it connected to? Itself, apparently, as Directory Access showed it as being connected to 127.0.0.1, which is what it should be set to if it’s a master. For some reason the server had somehow relinquished its role as master and handed it over to none other than itself. Talk about schizophrenic. Now, I have no idea — nor time to figure out at this point — how all this role stuff is managed down in the guts of the system. In fact, with the semester looming and all the replica problems I’d been having, I took this as a sign that maybe it was time to rebuild the server. And that’s what I did.

Mac OS X Server, for all its relative ease-of-use, sure is amazingly finicky about a great many things when it comes to installation and initial setup. And once it’s set up wrong, it seems the only way to reset all its various databases and services is to wipe it clean and start from scratch. There are settings (like the IP address) that get so ingrained into every database and config file that it’s nearly impossible to find every instance and change it. Kerberos, the LDAP database, and UNIX flat files all need to accurately know about things like the IP address and the Fully Qualified Domain Name of the server before you start any services. And DNS had better be working perfectly somewhere on the network you’re OS X Server is on, and it better have a perfect record for your server or you’re going to just be screwed. Maybe not so screwed that you can’t fix the problem, but probably screwed enough that rebuilding your server from scratch is the preferable option. I believe what caused all my problems — and I haven’t proven this yet, but I believe it to be so — was a dot. That’s right, a dot. One, single, stupid little dot I forgot to enter in my reverse DNS table. You know the one. You’ve surely forgotten it yourself at some point. The entry should look like:
34.1 IN PTR server.systemsboy.domain.com.

But instead you wrote:
34.1 IN PTR server.systemsboy.domain.com

Ooops! See the difference? No dot. No period at the end of the sentence. Kids, punctuation counts in DNS. And DNS counts on Mac OS X Server. Big time.

So now you’re fucked and you don’t even realize it. You go and you build your server, and things seem fine, and then all of a sudden the simplest of things — building your replica — suddenly don’t work, and you’ve no idea why. There are no error messages, no warning dialogs. Things just don’t work.

There were some other facts that I learned about over the course of this saga as regards the finickiness of Mac OS X Server. In Tiger Server 10.4+, f you want to host a replica, Kerberos has to be running properly. The Apple folks will tell you that once you set your server to “Open Directory Master” Kerberos will just set itself up and start automagically, which it will, as long as your DNS is set up right. Otherwise it will not do anything, and, as per the manual, you can start it up later, by hand, in Server Admin. But if you’ve already gone and done a bunch of other Kerberos-dependent stuff on your server — stuff like adding users, or creating a replica — and you then try to Kerberize your server, you’re most likely quite fucked. And there’s no easy way out. You’re best bet is to rebuild the server from scratch. The alternative is to scour the numerous databases and config files in an attempt to correct the problem. But you’ll never be sure it’s right. So you rebuild the server.

I rebuilt the server.

It wasn’t so bad. I’m getting really good at batch importing users into Open Directory (which will be the subject of a later post). The worst part was that the Windows machines all had to be rejoined to the new server by hand. Otherwise, with a proper DNS configuration — and, just to be on the safe side, with the master and the replica added to the /etc/hosts table — rebuilding the server was as easy as it’s supposed to be. And now I have a server in which I have a great deal more confidence. Kerberos, in particular, seems to be working properly now. The weird error messages and inability to authenticate via Kerberos using the OD admin user are finally gone.

If I may make an aside to wax bitchy here for one moment: It seems to me that Apple needs to make a much bigger deal of all these dependencies. Proper server operation — especially Kerberos configuration — is heavily reliant on proper DNS configuration. I’ve known that for a while, mostly from forums and admin sites rather than from Apple’s server documentation. But also, replication is dependent upon proper Kerberos configuration, which I never knew until I scoured the forums. I don’t even use Kerberos, so I never worried about it being configured properly. And back in Panther you could easily replicate without it. That’s changed and it would be nice if someone hollered about it quite loudly in the manuals. Also, it would be really great to have a way to put everything back to spec. FQDN assignment is done automatically by the server, and you no longer have the option to do it by hand. This happens quite early in the boot process and is completely dependent upon proper DNS. But if you screw up your DNS, there appears to be no safe way to go back and redo the FQDN assignation. Everything gets hosed by one little mistake. I’d like a button that resets everything on the server — Kerberos, FQDN, IP address and OD Database — back to the settings just after a clean install, and then brings you to the Setup Assistant to redo all the settings. At least then you wouldn’t have to reinstall everything. (You do realize, of course, that there is no “Archive and Install” feature
for Mac OS X Server. So any install requires the multi-megabyte software updates, and the reinstallation of any users, shares, computers and the like. Which is a big, fat, nasty pain.)

Okay, so the burning question: What about the replica? Did the rebuild yield a usable replica? The answer is not pretty: I think so.

Once I had my master server rebuilt (and cloned for easy restore!) I tried the replica. Replication seemed to go fine, but then, it did before too, so that wasn’t necessarily reassuring. Once the replica was built I pulled the plug on the master. This time, things were even worse than before. No authentication was able to take place on the clients. They just didn’t see the replica. But why?

One of the things I learned through all of this is that it’s the client who maintains the list of replicas. It gets this list from the master OD server when it binds to said server — i.e. when you open Directory Access and set it up — and it probably also refreshes the settings here periodically, perhaps when mcx cache is refreshed, but I’m not really sure. You can see all these settings in one particular preference file, called DSLDAPv3PlugInConfig.plist, located in /Library/Preferences/DirectoryService. One of the things you’ll see in this file is the IP address of your master server, and if there are any replicas, they will be listed as well. Checking this file on my clients showed that they had not gotten the replica info from the new server. They had no idea about the replica. So I plugged the master back in, and re-bound a client machine. It got the replica info. I unplugged the master again, and after a few minutes, my client could again authenticate to the replica. It worked! Finally!

Now all of this occurred on Friday at around 9PM. I’d worked about 52 hours by that point. Testing the replica on a larger scale, and on Windows in particular, would have required effort I was no longer capable of nor willing to exert — i.e. un-joining Windows systems from the master, removing them from the computer list, rejoining them, pulling the plug, testing the behavior, and the like, all of which would have taken at least another hour or so to do properly. And whether it worked or not, I’d still have to go rebind every machine one by one the following week, just to be safe. So I quit. And that’s why, when you ask me, “Did the replica work?” I can only answer, “I think so.”

To qualify: I think we have a winner. It seems to work on the Macs, and I think it will work on Windows (though how well it works on Windows, and if it works like we hope it does is anyone’s guess). The previous difficulties seem to have been caused by Kerberos-related problems on the master OD server, but I haven’t thoroughly tested it yet. And if it doesn’t work, then that’s that. We’ll probably give up on the whole replica idea for the semester, since the only way to test it is to effectively bring down the network, and I’m not so willing to pull plugs when school is in session. So next week I will test a Windows machine if I can find a good time to do it. Either way, I will be rebinding all Macs and Windows machines to the new server in the hopes that replication is indeed working. But if there’s no time to test it, we may not know for sure until the master server dies. Sure. It’s a little scary. But I’ve seen worse.

One last thing: I did have the strangest problem with my latest master/replica setup during testing. After pulling the plug on the master, and then plugging it back in, I was unable to authenticate to the master as the directory admin user from WGM, Server Admin or ARD. I was able to ssh in, but I could not reboot the machine from the shell because sudo from both the directory admin account and the local admin account would not accept the root password. I was totally locked out. The only thing I could do was hard reboot the server by holding the power button on the machine. Fortunately, after doing so, the server began behaving normally again, and root access was again right with the universe. Sure did make me nervous though.

Anyway, thus ends, essentially, the saga of our replica, at least for now, as well as, for the most part, the saga of Three Platforms, One Server. We’re now in the final stage of the project: going live. Class begins on Tuesday, and we’ll get to see how our master authentication server holds up under load. If disaster ensues, expect more posts on the subject. If all goes well — knock wood with me, people — I will post a final round-up article and this series will be concluded. Also, if I get a chance to more thoroughly test my new replica I will post the results either here or in another article.

Okay, everyone. Have a great school year!

In a few places along the travails of this series I mentioned the need for a fail over in the event our master authentication server goes belly up. The rationale is that, with all our eggs in the one server basket, if our master goes down, no one can log in — not on Windows, not on Mac, not on Linux. Fortunately, Mac OS X Server allows for what’s known as a replica. A replica is a read-only copy of your Open Directory data hosted by another server. But it’s a bit more than that. The replica also provides what’s known as fail over. That is, if the master server goes down, the replica knows to take over and start serving authentication to the clients. The replica, in effect, becomes the master in the event of the master’s absence, until said master returns to service, at which point the replica gracefully hands control back to the master. You can actually have multiple replicas for fail over, and for redundancy in the event of slow or separate networks. Brilliant! I’ve wanted one for a long time. And now I have it.

Setting up a replica is easy as pie: Set the server’s “role” to “Replica”, point at its master, authenticate and you’re off to the races. That is, if you’ve set everything just perfectly on both your master server and the replica. That’s the thing about Mac OS X server, and always has been: if you set it up wrong initially, you’re in for a potential world of hurt. As I rediscovered last week.

After building my replica, I next went on the requisite testing spree. The test involved physically pulling the network cable from the master server, and then observing the behavior of the clients. Initially, the replica seemed to do a fine job of picking up right where the master left off. After about two minutes’ time, clients could log right back in. But after about 5 minutes’ time, nearly every client on our network beachballed indefinitely, and any attempt to login would hang the machine. This hanging was so sudden and so thorough, it actually froze my machine mid-cube-effect as I attempted to fast-user-switch to the login window. Some fail over! Not cool.

Fast User Switch: Frozen in Time
(click image for larger view)

So I’ve spent a great deal of time figuring out the solution. The logs were no help. Google turned out to be a wash. Manuals? Pfft! For the first time in quite a while, it was an Apple Knowledge Base Article that offered the fix. The article was written for people who were having problems getting a 10.4.2 replica to remain a replica. Apparently there was an issue that involved these servers reverting back to “Standalone” roles after being switched from “Replica” to “Standalone” and back again. Though this was not my problem, nor did any of the symptoms reflect what I was seeing, I finally decided to try the recommendations in the article as they seemed fairly universally applicable, and as I was desperate and had tried everything else. The article essentially details methods for cleaning out every part of the master database that references the replica, and then re-promoting the replica to a clean master. Honestly, my master database looked pretty clean to me, but there was one bit of advice that I was not aware of, and it’s my suspicion that this was what did the trick for me: The OD Administrator of the replica’s database can NOT have the same UID or short name as that of the master. The article recommends creating a separate OD Admin account on the master, and using this separate account when binding the replica to the master. Honestly, I had no fucking idea. Would’ve been nice if this had been more explicitly mentioned in the manuals. Believe me, I’ve read the section entitled “Setting Up an Open Directory Replica” numerous times by now. It’s not there. Fortunately, it’s in that article, and now I know (and knowing’s half the battle).

And now you know.

So, in a couple weeks we go live with our unified internal network. I’ll let you know how it goes.

UPDATE 1:
Actually, my replica seems to still not be working, at least not very reliably. What happens is that about two minutes after the plug is pulled on the master, the replica picks up. At this point, clients — both Mac and Windows — can successfully log in. Shortly thereafter — maybe four minutes later — we start having problems: the Macs can’t log in, or only some can log in; Windows machines log in intermittently — one time it works, the next time it doesn’t, it works after a reboot, then it doesn’t; and, perhaps strangest of all, network connections to the Macs — ARD, ssh, anything but ping — become all but impossible, hanging at the attempts. This is totally a guess, but it seems to me like the clients are having serious trouble binding to the replica. They keep attempting to do so, with some initial or intermittent success, and in their attempts network connections get locked up and the machines bog down. It’s almost like the replica server is saying, “Yes, you can bind to me,” and then changing it’s mind and saying. “Wait, no you can’t. Never mind. Screw you.” Again, I’m only guessing. There is nothing clear cut in the logs, and I can’t find anything in Apple’s Discussion forums or Knowledge Base that specifically addresses my problem. I only pray that it isn’t a problem with my master server, but the master works perfectly, and it seems to me that a replica of a perfectly working master should work perfectly. The current replica is running on a Mac mini with limited RAM, and a 10/100 BT NIC, and I want to rule out potential problems that might be caused by the hardware as well as the software set up. So my next step will be to build a new replica from scratch on a G5. I’ll let you know if that solves the problem.

Another thing I absolutely should mention: For Windows replication, the replica server must be set up as a Backup Domain Controller (BDC). This is done in the Server Admin application in the Windows section. It’s fairly straightforward to set up, so I won’t go into detail, but just for the record, it’s important to be aware of this, and I wasn’t until recently, so I mention it here for completeness’ sake.

Having this replica isn’t absolutely critical to our plan. That is, we can go forward with this plan without the replica. But having a working replica will provide an important safety net that I’d really like to have working as the semester begins. There’s no good way to test it while the semester is in session. So I’m working hard to get it up and running in the final week of the summer.

So much for this post being short. More to come.

UPDATE 2:
I built the new server today. From scratch. On a G5. No joy. I honestly don’t know what the problem
could be. I can only guess that something either broke with the latest 10.4.7 update, or that there’s something slightly off with my master server and it’s causing problems on the replica. But it’s weird, because if I bind directly to the replica using Directory Access it works perfectly, which leads me to suspect a problem on the client. But it affects Windows machines as well, so that doesn’t quite figure either. I hate to admit it, but I’m stumped. And, unfortunately, I don’t have time to worry about it right now. I will revisit this issue at a later date, when I get some time. When I do, I’ll probably post a new entry with the solution, that is, if I’m able to find a solution. I hate this kind of thing. Is anyone else having a similar problem? I feel like I’m going nuts. And I can’t believe I’ve spent so much time on something that should be really, really easy. Fuck. What a bummer.

UPDATE 3:
I’ve pretty much given up on this for now. No time. And no good time to test, what with the students coming back next week. But today I noticed something strange, and it occurred to me it might be related to my replica problems. Today, when trying to make an AFP connection to the master server from a client using a simple “Connect to Server…” I got a Kerberos prompt that refused my admin credentials. Hmmm… Kerberos problems… On the master… Not good… So who knows? I may be rebuilding the master server at some point. But not now. Oh, Lordy, not now.

Today we discovered that applications to be used by network users on Windows machines must be installed by a network user, and this network user must be a domain admin. Put another way, if a Windows application is installed by a local administrator, it will not be fully accessible by users who authenticate via the domain hosted by the Authentication Server. Typical.

The solution is fairly easy, albeit kind of a pain. On each client workstation, you must give the network admin user (i.e., a user whose account exists only on the authentication server) full access to the “C” drive. You heard me: On each computer. Giving full access to a user on Windows requires logging in as a local admin and granting the network user full access rights. Windows then changes the access permissions on every file on the machine, which takes a long-ass time. You heard me: long-ass.

If we had to do this on every machine by hand, I’d be grumpy about it (but I’d do it anyway). Fortunately, we’ll be building our Windows boxes from clones, which means we only have to do this once on a master build. the change should propagate via the clones after that. So I’m a pretty happy camper. And we’re back on track.

Still, Windows, what the fuck?

And just to be clear on this, I honestly don’t know enough about Windows or Active Directory to understand why this is happening. All I know is what we’ve observed, which is that Windows apps fail to run properly for network users when installed by local users. We tried matching the users and groups we’d had on the old Windows Server, but that did no good. Setting full access locally for a networked user was the only thing we could find that worked. It’s very possible, however, that there’s a better solution than the one we’re using. If anyone can enlighten me, I’m all ears.

Oh, and yes, our network user has full access privileges for the directory domain. Confused? Me too.

More to come…

UPDATE: I found a better solution. A network user can be added to the “Administrators” group via the Users and Groups control panel. Doing so is not particularly intuitive, but it works and saves us the trouble of having to modify permissions on the entire “C” drive.

Windows “Advanced” Users and Groups: By “Advanced” I Think They Mean “Annoying”
(click image for larger view)

It essentially requires logging in as a local Administrator, navigating to the “Advanced” window in Users and Groups, and then adding the user to the “Administrator” group.

Group Properties: Do We Really Need This Window?
(click image for larger view)

Windows doesn’t see a network user unless she’s logged in, so you have to know how to enter a network user. It should look something like this: DOMAIN\user, where “DOMAIN” is your Active Directory Domain, and “user” is the user name.

Add the User Here: Who’d a Thunk It?
(click image for larger view)

You can find more complete instructions here, which is where I got these images.

Fun stuff.

For the Windows quota requirements we went with the first solution mentioned in this article, and outlined in detail here: Windows roaming profiles are on a separate, quota-enabled volume on the authentication server. This — and the Windows implementation in general — is my biggest concern, and will require a great deal of testing. Summer Session has begun, and this will give us the opportunity to test on live subjects as a limited pool of students begins using the systems for work again.

There are a couple of immediate benefits of this new system. For one, users can now change their password lab-wide from any Mac in the lab. Now, users don’t tend to do this much, but they can if they want to, and it’s easy as pie. But as an admin, the most exciting benefit to all this is the ease with which new users can now be created. In the past, creating a new user was a multi-step, multi-computer process that involved coordination between multiple sysadmins: Admin 1 gets the user info, creates the Linux and Windows accounts and hands it off to admin 2, who then creates the Mac account and uploads the Mac skel. This process was lengthy and extremely error-prone. Using the new system, user creation is a breeze. One single, easy-to-use script on the authentication server pretty much does it all. Any sysadmin — or even a trained assistant — can do it. Just log in to the authentication server, run the script, and you’re basically done. It’s fast, easy and accurate. Which is really the whole reason we’re doing all this. I’m pretty happy about it.

There are a few minor details I need to work out. The main one is file sharing. In addition to being an authentication server, our original Mac server is also a file server. On it there are numerous shares for various purposes — one for staff, one for students, etc.. At this point I’m thinking of keeping the file server and the authentication server separate. This will reduce the load on both, and mitigate the need to migrate any data. The catch is that user authentication data on the file server will need to match that of the new authentication server. This should be a simple matter of changing the old server’s role to “Connected to a Directory System” and pointing it at the new authentication server. I’ve never done this though, and it’s complicated by the fact that the old server is running Panther whereas the new one is running Tiger. Probably the best thing to do will be to wipe the old server (yes, I have a backup) and put Tiger on it, then redo the shares. But I’ll probably test with Panther first, just to see what happens. I’ll let you know.

In any case, this is, again, a minor problem that shouldn’t affect the overall plan much and should be relatively easy to figure out and implement. It’s just one of those little things you forget about when you’re drawing up your master plan for world conquest. “Oh yeah… What about the file server?…”

Otherwise, the conversion is on and seems to be going smoothly so far. Once we get into serious testing I’ll post the results. Hopefully this will all be done in the next few weeks and the conversion will finally be complete.

I won’t go into great detail about the problems and various solutions to the Windows roaming profile issue. I’ve already written plenty on it, and the previous posts outline it all fairly well. I will say that the intended solution was to provide Windows roaming profile quotas by setting them locally on each workstation. But, last week, as we moved forward with the plan, one of my fellow sysadmins, who is far more capable with Windows than I happen to be, pointed out the fact that certain applications (i.e. Photoshop, Maya, etc.) need a certain amount of disk space for temp files and what-not in order to operate. Setting small local quotas effectively keeps these applications from running properly.

We are currently testing a few other scenarios in which quotas for Windows roaming profiles can be implemented to our satisfaction:

1. The Authentication Server (Mac OS X Server 10.4.6) has a separate volume for Windows roaming profile storage and that drive has quotas enabled (this was the original plan). The drawback to this is that the user’s home account data is stored separately for Mac and Linux — which keep this data on the home account server — than for Windows, which would store this data on a reserved volume. The other drawback is the fact that the client machine is not aware of the quotas until the user logs out and the Windows client attempts to upload the new data, at which point Windows issues an error if the user has exceeded his quota. But the user is not warned about quota violations until logout, and this could cause some minor problems.
2. The Windows Server continues to host roaming profiles and determine quotas for Windows users, but gets user authentication information through a connection to our Mac Authentication Server. The drawback here is that we have to continue using the Windows Server, which we don’t really want to do, though it does seem to give us a slightly higher level of control than does the Mac Server. This method is also complicated by the fact that we are currently running Windows Server 2000, which does not include native authentication to LDAP, so third-party solutions would be required. This method could also complicate user creation.
3. Through some combination of Windows and Mac Servers, we convince the Windows roaming profiles to be situated on the Temp volume of the local workstations, rather than the default location on the “C” drive in the “Documents and Settings” folder, and then set quotas for the Temp volume. I’m skeptical that this is even possible. Windows seems to be hard-coded in ways that make specifying the location of roaming profiles anywhere other than “C:\Documents and Settings” impossible. So this last option seems the least likely to succeed, though if it did it would match the way Mac and Linux behave much more closely. And if we could figure out how to do this without the Windows server, it would be almost ideal.

(Actually, I just thought of a problem with this last method: The Windows Temp drives get erased every Friday. If users happen to be working during the deletion period, what happens to their roaming profiles? The same thing happens on the Mac, but the deletion script on Mac does not delete work owned by the currently logged in user. Can such a scenario be implemented on Windows?)

Most likely we will go with the first solution. We already know that it works. It’s a little extra effort when creating new users, but that’s totally scriptable. We plan to do user creation from a single script from now on anyway, so these extra few steps, once incorporated into our user creation script, won’t really be a big deal at all. The only other problem with this is that our replica will now need to sync the Windows roaming profile volume as well if it’s to work as a proper fallback system. This, too, should not be terribly difficult to accomplish. Overall, this solution is less elegant than the original one, but it should be workable. Hopefully, Windows Vista will mitigate a lot of these problems. (Yes, that was totally a joke. Please chuckle softly to yourself and move on.)

I guess what amazes me still is how contrary to other operating systems the Windows OS is. Everything we’re doing here can be done the same way on Mac as it can on Linux, or virtually any other *NIX system: home accounts can be read directly from a network disk, their locations can be specified, and therefore, all this quota nonsense is unnecessary. On Windows, roaming profiles apparently must be downloaded to the client machine (an unbelievably stupid requirement), and the location of said profiles apparently must always be on the root drive in the “Documents and Settings” folder. I guess these are the ways in which Windows continues to force people to use Microsoft products (I can almost hear Bill Gates whispering in my ear, “Wouldn’t it all be easier if you just used Windows Server?”) But for software that’s become the dominant standard in both the business and personal markets, Windows sure seems non-standard in baffling and infuriating ways. Though this may be how Microsoft has managed to stay on top all these years, I still, perhaps naively, believe that some day, if they don’t change this strategy, it will hurt them. Frankly, though I’m sure they’re out there, I don’t know a single sysadmin that likes Windows. Can you blame them?

The general outline of what I’ll be doing over the next few days goes something like this:

• Backup my current Mac server (for safety)
• Build my master authentication server
• Backup a clone of the clean server install
• Configure the new server with:
• Users and Groups
• Home account automounting
• Home account sharing to SMB (for Windows Roaming Profiles)
• A skel account for Windows users (to live on the home account server)
• Other share points
• Create a replica of the new master server

What I did today:
Well, it’s amazing how long a base install of Tiger Server can take. I’ve pretty much been doing that all day. Not that I’m so incompetent that I can’t install the software in seconds flat, but software updates take forever and a day. Planning and getting drives to do all this on was a bit of an effort too. Plus I just wanted to make sure I did it right the first time, so I went slow, gave myself the day. I’m also making clones of everything along the way, for building my replica, and in case I goof and need to start over. So that takes a while. I guess I’m just saying that I’m taking my time with this, ’cause I want it to be as perfect as possible from the get-go.

By Monday we should have:

• A base install of Tiger Server 10.4.6 with requisite Software Updates on a firewire drive
• A backup of our old Mac Server
• A new Tiger 10.4.6 authentication server that’s configured to host Mac, Windows and Linux users
• A replica of same

We will spend part of next week pointing all our workstations at the new server. The Windows machines will be the biggest pain as 1) they are running Windows, and 2) they need local quotas set (which could really be just a subset of point 1, but whatever). The reason for all this quota nonsense, you ask? Well, for the answer, you’ll just have to read the previous posts on the matter. Suffice to say, I’m hoping the quota setting nonsense will be the worst part of this job, which it should if all goes according to plan, which, I’m sure you’re aware, it rarely does.

Finally, I wanted to mention this quote that I read on Daring Fireball, by someone called John Gall, author of Systemantics, as it really jives with a lot of the stuff I’ve been thinking about with regards to the lab:

“A complex system that works is invariably found to have evolved from a simple system that worked….A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over, beginning with a working simple system.”
— John Gall

Next week should be interesting. I’ll keep you posted.

Servers are made to handle lots of different tasks and to keep running and doing their jobs under extreme conditions. To a certain extent, that is the very nature of being a server. To serve. Hence the name. So servers need and tend to be very robust. Nevertheless, they do go down from time to time. That’s just life. But in the world of organizations that absolutely must have constant, 24 hour, ’round-the-clock uptime, this unavoidable fact of life is simply unacceptable. Fortunately for me I do not inhabit such a world. But, also fortunately for me, this notion of constant uptime has provided solutions to the problem of servers crashing. And while I probably won’t lose my job if a server crashes periodically, and no one is going to lose millions of dollars from the down-time, no SysAdmin likes it when he has to tell his users to go home for the night while he rebuilds the server. It just sucks. So we all do our best to keep key systems like servers available as much as possible. It’s just part of the deal.

So how are we going to do this? Well, one of the reasons I decided to use a Mac for this project is that it has built-in server replication for load balancing, and, yes failover. We’re not too concerned with the load balancing; failover is what we’re after. Failover is essentially a backup database that is a replica of a primary database, and that takes over in the case of a failure of the primary database. Mac Server has this built-in, and from what I read, it should be fairly easy to set up. Which is exactly what we’re about to do.

The first thing we need is our primary server. This is the main server. The one that gets used 99% of the time (hopefully). We have this (or at least a test version of it) built already as discussed in Part 1. What we need next is what is called the replica. The replica is another Mac OSX Server machine that is set to be an “Open Directory Replica,” rather than an “Open Directory Master.”

So I’ve built a plain old, vanilla, Mac Server, and set it initially to be a Standalone Server. I’ve given it an IP address, and done the requisite OS and security upgrades. (Oy! What a pain!) In the Server Admin application, I set the new server to be an “Open Directory Replica.” I’ll be asked for some information here. Mainly, I’ll need to tell this replica what master server to replicate. Specifically I’m asked to provide the following at the outset:

IP address of Open Directory master:
Root password on Open Directory master:
Domain administrator’s short name on master:
Domain administrator’s password on master:

(The domain administrator, by the way, is the account used to administer the LDAP database on the master.)

Once I fill in these fields I’ll get a progress bar, and then, once the replica is established, I’m basically done. There are a few settings I can tweak. For instance, I can set up secure communications between the server with SSL. But for my purposes, this would be overkill. I’m pretty much going with the out-of-the-box experience at this point. So for setup, that should be it. Setting up a replica is pretty easy stuff.

Establishing the Replica: Could it Be Any Easier?
(click for larger view)

Now here comes the fun part: testing. What happens if our primary server goes offline? Will the replica take over authentication services? Really? I’d like to be sure. What I’m going to do now is test the behavior of the Master/Replica servers to make sure it acts as intended. The best way I know to do this is to simulate a real-world crash. So I am binding one of my clients to my Master server, with Replica in place. Then I’m going to pull the plug. In theory, users should still be able to login to the bound client. Let’s try it…

Bang! It works! I’m a bit surpsrised; last time I tried it, years ago, it (or I) failed. This time, though, it worked. We bound a client to the Master, our mother-ship server. Authentication worked as expected. (We knew we were bound to the new server because the passwords are different.) And then we killed it. We killed the master and logged out. There was some beachballing at logout. But after a few minutes — like two or three, not a long wait at all — we were able to complete logout, and then log right back in as though nothing had happened. I tell you, it was a thing of beauty.

So let’s briefly recap where we’ve been and what’s left to do.

Where we’ve been:

• We’ve built our Mama Server. Our authentication server for the entire lab.
• We’ve figured out how to migrate our users to Mama, and how to handle the required password change.
• We’ve solved the inherent problems with Windows clients and figured out a few solutions for handling them involving quotas and various roaming profile locations.
• We’ve built and tested the operation of the Open Directory Replica, and it is good.

What’s left to do:

• Well, honestly, not a whole Hell of a lot.
• The next step, really, is real-world testing. We have a basic model of how our servers and clients should be configured, and it’s basically working. To really test this, we’ll need to take some actual clients from the lab and set them up to use the new system.
• Stress testing (i.e. seeing if we can break the system, how it holds up under load, etc.) would also be good, and might be something to do over Winter break a bit, and definitely in the Summer. To do this, we’ll need to set up several client systems, and get users (guinea pigs) to do some real work on them all at the same time.
• Once stress testing is done, if all is well, I’m pretty sure we can go ahead and implement the change. I can’t foresee any other problems.

So I’m at a stopping point. There’s not much else I can do until the break, at which point I’ll be sure and post my test results.

Hope to see you then!

Doing this also prevents the error message at logout due to exceeding a server-side quota. In this new, local-quota scenario, the user is alerted the instant he exceeds his quota and can rectify the situation immediately. But if he doesn’t, no problem. As long as the local quota and the server-side quota match, he’ll never run into the problem of being unable to save his settings to the server, since he’ll never be able to exceed his quota anyway.

It turns out that, unlike on Mac OS X, setting quotas for NTFS volumes is incredibly easy. In fact, it’s a simple matter of viewing the properties of the drive on which you want to set quotas.

Windows Quota Settings: Easy as Pie
(click for larger view)

Here, go to the Quotas tab and click “Enable quota management” and “Deny disk space to users exceeding quota limit,” set the limits, and you’re basically done. The administrator will have no quotas, and any new user will have the quotas specified in this panel. You may, however, want to set certain users (particularly other admin or local users) to have larger quotas, or none at all. Again, exceptionally easy: Click that badly named, but ever-useful little “Quota Entries…” button and you’ll get a list of local users and their quotas.

Windows Quota Entries: Edit Specific User Quotas
(click for larger view)

Here, you can set quotas for specific users. Initially, you’ll only see local users. But after any network user logs in, you’ll also see them listed as well.

The more I think about it, the more I like the idea of local Windows quotas. Using them does away with all the problems mentioned in earlier posts, and may help with a lot of the quota-related problems users have with our current, Windows-server-based system. Also, this would allow me to store all profile-related stuff in the same place for Windows as for Mac and Linux — on our home account RAID — as I’d wanted to do in the first place. And, in general, it should be much easier — or at least not too difficult — to maintain.

A last note on the timing of this project: I just spoke with my boss about the fact that I’m way ahead of schedule and have been thinking about implementing our unified password server over the Winter break. Lately I’ve had some misgivings about doing this, and he echoed those sentiments. His take was basically, “If it ain’t broke, don’t fix it.” Always sage advice. And as gung-ho as I am to see this working, giving it some time and implementing it over the Summer will give me more time to plan this more carefully and test it more thoroughly, so that we can iron out problems at the beginning of the school year — when students’ work is not quite so in-progress — rather than at the end — when students are trying to finish their theses. This seems like a wise approach, and at this point I’m leaning toward erring on the side of caution.

Finally, credit where due: In-depth information on NTFS quotas can be found on this Microsoft page, which is where I lifted the images in this post ’cause I’m too lazy to figure out how to get good screen captures in Windows, and ’cause those images described perfectly what I wanted to show. I believe the images are in the public domain and, therefore, legally usable in my post, but if I’m wrong, I apologize, and if I need to take them down, someone can let me know in the comments section.