greyduck.net

Following up on the excitement of the other night, here’s some more fun and frolic at Ye Olde Office:

• Remember how the Snap! server died during the power glitch? That was my main backup device, followed by tapes (which I trust much less than I do a nice set of spinning drives). I now have to run tape backups directly off of the main fileserver. There’s no live standby from which to easily pull files, and I have to hope that the tapes themselves are okay. I’ll have to start doing partial restore tests I suppose… when they give me the time. And heaven forbid somebody important needs something important restored from those tapes. Bleah.
  
• Guess what I found out today? The version of Tapscan (the main sales software package we use) currently installed has a major glitch, and the new version won’t be ready for months! I get to spend the night at the station tomorrow! Yay! I’ll be going from computer to computer all through cubeville and through various sales managers’ offices and so forth, uninstalling the Tapscan client on each. Let’s see, 60-some-odd computers, that shouldn’t take more than… all night, right? Oh, but then I get to install an older version of the software and make sure it installs and runs okay! Yay, fun. Let’s not forget sticking around for at least a few hours the following morning to troubleshoot any major issues.
  
• I’ll be escaping from that hell midday on Friday after putting in what could be a thirty-hour day, only to return on Sunday morning, bright and early. Why? For another round of attempting to make correct AS/400 backups. We tried this last weekend with only partial success, and we must have a complete set of backup tapes ASAP before we can proceed with upgrading the server. Which, of course, they want to do ASAP.

And that’s just the three biggest issues that have come up this week. Don’t you wish you had a job as grand as mine?

I don’t remember much about the phone call I received at 12:30 this morning. I do know that at first I was completely mistaken about the time of day, who I was talking to and the topic of conversation. Give me a break, I had only been asleep an hour, and the day before that I had been running on about three hours’ sleep.

By the time Mike got to the phrase “all our stations are off the air,” I’d awakened enough to realize I was heading to the office as soon as possible.

Turns out that in the process of servicing the building-wide UPS, the electricians managed to cut power for the facility. All of it.

Every server, workstation and networked device had its power very, very interrupted.

Mike, CJ and I spent three hours running around the building trying to bring everything back online. I swapped out the fried power supply in the voicemail server, but didn’t have another spare to bring one of the production room PCs back to life. The Snap! server (remember that fine piece of equipment?) is completely shot to hell. This would make me happier if I hadn’t been trying to use it as part of my backup strategy, dammit!

One of the hard drives in one of the three “logger” computers died, which means that we’re losing out on stored audio from some of our stations. We had to cycle power on most of the Cybex KVM system not once but twice. The audio inventory database required rebuilding before production workstations could function again.

My Win2K workstation refuses, even now, to sign onto the Novell network. My Linux workstation appeared to lose one of its drives, but that turned out to be some kind of weird mounting glitch that I still need to investigate.

A number of servers had default-route problems that needed remedying, up to and including this very webserver (whose routing problem I didn’t think to fix until after I’d gotten some sleep). The tape backup machine is totally confused. I had to manually start the web-server process on Washuu. The access-control machine went nuts, but that’s actually an everyday occurrence. The email server required some TLC, as did the Groupwise API on the voicemail server.

We finally escaped at 3:30am, and I threw myself into bed and was asleep by 4:00. Only to wake up about six hours later and stumble in to the office to see how many annoying little problems simply couldn’t solve themselves in my absence… for instance, the kind that generate nasty emails about how “I can’t do my job until this is fixed.” (Gee, do you think the guilt-tripping and brow-beating is going to make me more inclined to help you? Huh?)

Why do I get the feeling that I should just stop trying to take vacation days?