I worked something resembling a normal day yesterday… 9 to 5, ya know. Today is different.

I arrived at 10:00am for a meeting (yes, I slept in, what’s your point?) and then put out small fires while waiting for the start of this evening’s AS/400 project: Upgrading hardware.

You see, there are four hard drives. They’re not large enough. So we ordered bigger ones. The trick to upgrading ’em is to move all the data off of one, replace it with a larger one, then move as much data as possible onto that new one. Then you repeat with another drive, and so on, and so on.

It’s 8:00pm as I write this. I don’t know how much longer this will take, since it could finish quickly or slowly depending on how quickly the machine can migrate data, and on whether any unforeseen difficulties arise. Even if all goes well I don’t expect to leave here until at least 10:00pm.

Here’s the kicker: I was at 33 working hours when I left yesterday. I’m now at 43 hours and counting, with an unfinished night’s project still to supervise and two more working days left in my week. Nevermind that I’m working another long shift this coming Sunday to match the one I pulled three days ago this past Sunday.

I need a rest. Among other things.

Here’s an example of the kind of day (and by “day” I mean “consecutive number of hours that may or may not number less than twenty-four”) I can have at work. And by “example” I mean “accounting of the last twenty-some-odd hours and the multitude of hours yet to come.”

8:00am Sunday: Arrive, greet consultant, turn him loose on project.

5:30pm Sunday: Leave consultant to own devices so as to procure foodstuffs and a few hours’ rest (and by “rest” I mean “hang out with friends, watch a movie, eat some din-din” but not “get some sleep”).

11:30pm Sunday: Return to office, send consultant home so he can get a few hours’ rest (meaning “sleep” this time) while I babysit the troublesome AS/400 backup.

3:00am Monday: Finish tape backup, go home to sleep.

10:30am Monday: Arrive, greet consultant, get marching orders for the coming evening, set auditors up with network hub and some cables, discover hacking of Windows 2000 server, repair damage, patch, patch some more, reboot some more, and start tackling all of the other small emergencies that tend to crowd up a standard Monday.

6:??pm Monday: Begin yet another tape backup of the AS/400, now expected to take four hours per session instead of the two hours forty minutes that it used to take.

?:??pm Monday: Go home, try to sleep enough to take on Tuesday…

Remember that power zap a week ago? Did I mention that Beast, the standby Enco server, has been misbehaving since then?

It’s not misbehaving anymore. It’s totally dead. Yes, I realize that this is a fine distinction.

So I’m going from dealing with the Win2K/XP RPC worm (it only hit Ryoko and Washuu, no biggie) to rebuilding the standby Enco server from the ground up. Did I mention that I don’t have a current backup of its Samba config or any of its scripts?

If you need me I’ll be in the other room, ruing the day I was born…

Following up on the excitement of the other night, here’s some more fun and frolic at Ye Olde Office:

• Remember how the Snap! server died during the power glitch? That was my main backup device, followed by tapes (which I trust much less than I do a nice set of spinning drives). I now have to run tape backups directly off of the main fileserver. There’s no live standby from which to easily pull files, and I have to hope that the tapes themselves are okay. I’ll have to start doing partial restore tests I suppose… when they give me the time. And heaven forbid somebody important needs something important restored from those tapes. Bleah.
  
• Guess what I found out today? The version of Tapscan (the main sales software package we use) currently installed has a major glitch, and the new version won’t be ready for months! I get to spend the night at the station tomorrow! Yay! I’ll be going from computer to computer all through cubeville and through various sales managers’ offices and so forth, uninstalling the Tapscan client on each. Let’s see, 60-some-odd computers, that shouldn’t take more than… all night, right? Oh, but then I get to install an older version of the software and make sure it installs and runs okay! Yay, fun. Let’s not forget sticking around for at least a few hours the following morning to troubleshoot any major issues.
  
• I’ll be escaping from that hell midday on Friday after putting in what could be a thirty-hour day, only to return on Sunday morning, bright and early. Why? For another round of attempting to make correct AS/400 backups. We tried this last weekend with only partial success, and we must have a complete set of backup tapes ASAP before we can proceed with upgrading the server. Which, of course, they want to do ASAP.

And that’s just the three biggest issues that have come up this week. Don’t you wish you had a job as grand as mine?

I don’t remember much about the phone call I received at 12:30 this morning. I do know that at first I was completely mistaken about the time of day, who I was talking to and the topic of conversation. Give me a break, I had only been asleep an hour, and the day before that I had been running on about three hours’ sleep.

By the time Mike got to the phrase “all our stations are off the air,” I’d awakened enough to realize I was heading to the office as soon as possible.

Turns out that in the process of servicing the building-wide UPS, the electricians managed to cut power for the facility. All of it.

Every server, workstation and networked device had its power very, very interrupted.

Mike, CJ and I spent three hours running around the building trying to bring everything back online. I swapped out the fried power supply in the voicemail server, but didn’t have another spare to bring one of the production room PCs back to life. The Snap! server (remember that fine piece of equipment?) is completely shot to hell. This would make me happier if I hadn’t been trying to use it as part of my backup strategy, dammit!

One of the hard drives in one of the three “logger” computers died, which means that we’re losing out on stored audio from some of our stations. We had to cycle power on most of the Cybex KVM system not once but twice. The audio inventory database required rebuilding before production workstations could function again.

My Win2K workstation refuses, even now, to sign onto the Novell network. My Linux workstation appeared to lose one of its drives, but that turned out to be some kind of weird mounting glitch that I still need to investigate.

A number of servers had default-route problems that needed remedying, up to and including this very webserver (whose routing problem I didn’t think to fix until after I’d gotten some sleep). The tape backup machine is totally confused. I had to manually start the web-server process on Washuu. The access-control machine went nuts, but that’s actually an everyday occurrence. The email server required some TLC, as did the Groupwise API on the voicemail server.

We finally escaped at 3:30am, and I threw myself into bed and was asleep by 4:00. Only to wake up about six hours later and stumble in to the office to see how many annoying little problems simply couldn’t solve themselves in my absence… for instance, the kind that generate nasty emails about how “I can’t do my job until this is fixed.” (Gee, do you think the guilt-tripping and brow-beating is going to make me more inclined to help you? Huh?)

Why do I get the feeling that I should just stop trying to take vacation days?