I don’t remember much about the phone call I received at 12:30 this morning. I do know that at first I was completely mistaken about the time of day, who I was talking to and the topic of conversation. Give me a break, I had only been asleep an hour, and the day before that I had been running on about three hours’ sleep.
By the time Mike got to the phrase “all our stations are off the air,” I’d awakened enough to realize I was heading to the office as soon as possible.
Turns out that in the process of servicing the building-wide UPS, the electricians managed to cut power for the facility. All of it.
Every server, workstation and networked device had its power very, very interrupted.
Mike, CJ and I spent three hours running around the building trying to bring everything back online. I swapped out the fried power supply in the voicemail server, but didn’t have another spare to bring one of the production room PCs back to life. The Snap! server (remember that fine piece of equipment?) is completely shot to hell. This would make me happier if I hadn’t been trying to use it as part of my backup strategy, dammit!
One of the hard drives in one of the three “logger” computers died, which means that we’re losing out on stored audio from some of our stations. We had to cycle power on most of the Cybex KVM system not once but twice. The audio inventory database required rebuilding before production workstations could function again.
My Win2K workstation refuses, even now, to sign onto the Novell network. My Linux workstation appeared to lose one of its drives, but that turned out to be some kind of weird mounting glitch that I still need to investigate.
A number of servers had default-route problems that needed remedying, up to and including this very webserver (whose routing problem I didn’t think to fix until after I’d gotten some sleep). The tape backup machine is totally confused. I had to manually start the web-server process on Washuu. The access-control machine went nuts, but that’s actually an everyday occurrence. The email server required some TLC, as did the Groupwise API on the voicemail server.
We finally escaped at 3:30am, and I threw myself into bed and was asleep by 4:00. Only to wake up about six hours later and stumble in to the office to see how many annoying little problems simply couldn’t solve themselves in my absence… for instance, the kind that generate nasty emails about how “I can’t do my job until this is fixed.” (Gee, do you think the guilt-tripping and brow-beating is going to make me more inclined to help you? Huh?)
Why do I get the feeling that I should just stop trying to take vacation days?