I don’t remember much about the phone call I received at 12:30 this morning. I do know that at first I was completely mistaken about the time of day, who I was talking to and the topic of conversation. Give me a break, I had only been asleep an hour, and the day before that I had been running on about three hours’ sleep.
By the time Mike got to the phrase “all our stations are off the air,” I’d awakened enough to realize I was heading to the office as soon as possible.
Turns out that in the process of servicing the building-wide UPS, the electricians managed to cut power for the facility. All of it.
Every server, workstation and networked device had its power very, very interrupted.
Mike, CJ and I spent three hours running around the building trying to bring everything back online. I swapped out the fried power supply in the voicemail server, but didn’t have another spare to bring one of the production room PCs back to life. The Snap! server (remember that fine piece of equipment?) is completely shot to hell. This would make me happier if I hadn’t been trying to use it as part of my backup strategy, dammit!
One of the hard drives in one of the three “logger” computers died, which means that we’re losing out on stored audio from some of our stations. We had to cycle power on most of the Cybex KVM system not once but twice. The audio inventory database required rebuilding before production workstations could function again.
My Win2K workstation refuses, even now, to sign onto the Novell network. My Linux workstation appeared to lose one of its drives, but that turned out to be some kind of weird mounting glitch that I still need to investigate.
A number of servers had default-route problems that needed remedying, up to and including this very webserver (whose routing problem I didn’t think to fix until after I’d gotten some sleep). The tape backup machine is totally confused. I had to manually start the web-server process on Washuu. The access-control machine went nuts, but that’s actually an everyday occurrence. The email server required some TLC, as did the Groupwise API on the voicemail server.
We finally escaped at 3:30am, and I threw myself into bed and was asleep by 4:00. Only to wake up about six hours later and stumble in to the office to see how many annoying little problems simply couldn’t solve themselves in my absence… for instance, the kind that generate nasty emails about how “I can’t do my job until this is fixed.” (Gee, do you think the guilt-tripping and brow-beating is going to make me more inclined to help you? Huh?)
Why do I get the feeling that I should just stop trying to take vacation days?
Comments
6 responses to “Swift (re)Boot To The Head”
*hugs*
nuf said
ROFL at what sean said. Sounds like a good plan to me. Especially since I was the one driving Karel to work and back.
Oh, you poor thing *huggles* That’s when you need a nice baseball bat….
See, I understand about half of that technical jargon, but I’m getting the idea that you’ve simply had a hellish day. Hang in there.
Prevention proprosal:
Step 1: Kill all the electricians and their family’s
Step 2: Rue the day they entered the building
Step 3: Rue again
Step 4: Double-check to ensure that indeed ALL family members are indeed, still dead.
Step 5: Rue one last time
It may not do much, but it’ll sure feel good.
The positive spin: someday this will all make great material for the Karel Biography on A&E.