First thing Tuesday morning, the main Enco digital audio storage server died. After hours of troubleshooting and tinkering, we brought it back online… minus all of its audio data that was lost when one of the drives in its array died. (We’re running RAID 0 because it’s the only way we can get enough capacity out of the array.)

We switched everybody over to the standby server, and told that server to restore its audio data to the main. No problem, right?

Wrong.

Shortly before midnight last night, the standby server decided to hang. We lost ten hours out of our restoration process because the server didn’t crash enough to come to the attention of my server monitor. Argh. (It was reporting X amount of available drive space… on a drive that was no longer responding. *grumble*)

The plot thickens: An attempt to restart the “rsync” file transfer process revealed another problem. Rsync was starting over from the very first file… even though no differences between the source and destination files could be determined! I’ve come to believe that using rsync across operating systems is a Very Bad Idea. (I’m having a similar problem backing up our main office server. Le sigh.)

So now I get to manually copy batches of audio data from the standby to the main server. This should only take, oh, another dozen hours or so. Nevermind that the copy command does a slower job than rsync, and nevermind the headache of updating to catch deleted files. I’m probably going to have to perform evil involving something like DirComp from a Windows workstation mapping both servers.

All of this adds up to no vacation day tomorrow, and I’ll probably be here Saturday and Sunday as well. I’m going to be gone Monday, no matter what: No way am I missing out on Two Towers!

UPDATE: It’s just about 9pm, and I’m just about to head back to the office. Why? Because the file transfer has stopped again. This graph shows the traffic on the network port through which the file transfer is being done. See that sudden stop right around 5:30? Right about the time I was leaving the building? Yeah, that’s just peachy.

Near as I can tell (from shelling in to the standby server from home), this time the stoppage isn’t because the RAID controller in the standby server is a piece of crap. (That was the cause of last night’s crash.) I don’t know why it’s stopped, really. I only know that I have to go back down there and start it up again. Le sigh.

At this rate I really will still be working on this “project” come Saturday. Argh. I love my job, really I do… but sometimes there are parts of my job I could do without.

It’s just about 7:00 in the evening, and I’m taking a dinner break before diving headlong into tonight’s project: Going from computer to computer in the building to remove the printer objects for the now-removed Savin copier/printers, and replacing them with properly-configured printer objects for the new Toshibas.

That’s right, today at noon the new copier/printer devices arrived. My entire afternoon and evening has been spent dealing with them… installing, testing, configuring, more testing, tinkering, more configuring, more testing.

Well, except for that fun hour when the corporate WAN went down. Thanks, Worldcom! I needed some more excitement!

Not being able to efficiently “push” the printers out to the network clients means a long night spent manually installing both machines onto each of some 60 computers around the building. Sometimes being a “Department Of One” really blows, you know?

See you in the morning, folks.

It was officially revealed today that Daria O’Neil, she of the former Gustav And Daria morning show, will start Monday on another morning show. On another Entercom station.

She’s joining Nelson and Terry on 105.1 The Buzz.

Sadly, this means that Teri Ann has been shown the door. So, once again we learn that you can’t get something for nothing. Y’all had better hope that Nelly, Terry and Daria-y can generate some real chemistry (and wow, did I belabor the -y thing or what?), ’cause I’m here as witness to the broken carcasses of failed line-up changes that litter the radio morning-show landscape. That’s not to say this is doomed to failure, and I know that Nelson and Daria both are funny and clever people.

I’m just, you know, kinda worried.

Anyway. Y’all can get your daily fix of Daria, starting Monday. Rejoice, and stuff.

(Oh, and props to some guy named Steve for reminding me about this. I was going to post this earlier today, honestly I was…)

Goddammit.

At about 3:15 this afternoon, our Groupwise server’s main mail storage volume ran completely out of disk space. All attempts to salvage the situation with the server code running failed utterly and miserably.

As of this writing I’ve finally managed to semi-gracefully reboot the server, check and mount its volumes, delete an unneeded installer directory from the volume in question, and start a purge of sent items, trashed emails and (most importantly) oversized emails.

I don’t know if this is going to work, mind you. Groupwise does funny things with purged emails, like (for instance) hanging on to them even though you’ve clearly told it you don’t want them anymore.

It’s going to be a long, long evening here at the office, folks. I’ll update this entry as (or more accurately “if”) the situation progresses… or regresses as the case may well be. *sigh*

And in case you’re wondering: No, I don’t have disk space alarms set on this particular server. SNMP troubles, doncha know. The sort of thing I’d have time to tackle if, say, I had an assistant… but we all know that’s a pipe dream, eh?

Wish me luck. Lots of it.

UPDATE: The first purge took 3 hours 20 minutes and brought the server from 99% full (I started the purge after deleting the client install directory so I’d have that one percent to work with) down to about 93%. The second purge took exactly one hour and brought the server down to 65%. How, you ask, did this miracle occur during the second purge? Easy. I was deleting any email larger than 1.5 megabytes. That’s right, folks. About 30% of a 34 gigabyte drive was taken up with very large emails, essentially emails with large files attached. Lovely.

There’s gonna be some pissed-off folks around the building tomorrow when they realize that their mp3s, videos and PowerPoint files are all missing out of the email system… where they didn’t belong in the first place.

Me, I’m going home now. It’s about damned time. Two 12-hour days in a row is not my cuppa, baby.

I unpacked our new GM’s computer from CDW, and instead of the usual unmarked keyboard box I found this:

Wow. Could we be a little bit more full of self-importance, please? Sheesh!

*smirk*

And in the “oh, these are the problems you wanna have” department, it turns out that I have to burn a whole week’s worth of vacation hours on top of what I’m already spending to do OryCon and take an extra day after Thanksgiving weekend. Since I can’t take an entire week off or Very Bad Things will happen, it looks like I’m going to be taking a whole slew of three-day weekends…