Let me tell you how the sequence of events went down. (“Down” being the operative, or inoperative, word.)

1) Something Happens at about 5:30 this morning. (All times Pacific, thank you.) Perhaps one Audicy session too many was opened, perhaps it was just that the server had had enough. Either way, the Audicy server and (probably) the two workstations began flooding the network with traffic. Also, the Audicy server’s root partition is chock-full, which is likely the cause of the commotion (as the clients madly try to read and write data on the server).

2) A bit after 6:00, my phone beeps to let me know that we can no longer communicate with Entercom Corporate over the WAN. This happens fairly often, and usually clears itself up, so I fail to panic and instead go back to bed.

3) A bit after 7:00, my phone rings. It’s John Graefe, my boss at Corporate, asking why our WAN router is incommunicado. Whoopsie. I tell him I’ll jet right down to the office and Deal With It.

4) By 8:00, I’ve burped the WAN router to no avail.

5) At about 8:30, I notice that the main office network switch is saturated. Instead of happily blinking busy little activity lights, the whole display is lit solid. This worries me. I ponder burping the switch, but that would kick everybody in the building off the network, causing all kinds of chaos. Instead I start pulling plugs, briefly, one by one until the lights go dark(er).

6) I discover that unplugging the cable that connects to the Audicy server makes Everything All Better. Upon investigation of the server, I see that the root partition is full. Unloading the Netware emulation code frees up space. I decide to reboot the machine. This is great, but Linux insists on checking all the partitions since the machine hasn’t been rebooted in ages.

7) While the 60-plus gigabyte partition is being slowly fsck’d (and that’s a technical term, thank you), I look in horror at the central switch, which has again lit up solid.

8) After unplugging a series of cables around the server room, kicking half the building off the network in the process, I discover that a machine in Production Room 3 is the culprit. Hmm, what’s in there? Oh yes. An Audicy workstation. The workstation in question is powered down, which makes Everything All Better Yet Again.

9) I check the Audicy server, all seems well. For grins and chuckles (since both workstations are turned off) I upgrade the kernel from 2.4.0-test10 to 2.4.20, mostly in the vain hope that the memory/space leak will go away.

10) The Audicy server is rebooted on the new kernel, and the Production Room 1 workstation is brought back online and tested for network connectivity. All is well. The Production Room 3 workstation, already due to be removed since there’s now a ProTools workstation in place, is completely disconnected pending physical removal from the room.

There’s my morning in a nutshell… an appropriate receptacle. Now, if you’ll excuse me, I’m going to attempt to rehabilitate the general-purpose computer in Production Room 5, which seems to have suffered a catastrophic failure for unknown reasons…

Dr. Doug and Skippy emailed me this afternoon asking me to look into the ongoing printer problem in the Rosey studio. “Sure thing,” I said, “I’ll just pay Sheryl a visit.”

I went in, looked at the printer, power-cycled it and was about to run a test print when Sheryl Stewart said, “Hey, I need you to read something.”

“Uh, I don’t read very well.”

“That’s okay,” she replied, “It’s supposed to sound bad.”

“Okay, then.”

And so I took part in an on-air contesting bit. The evidence is right here. (Nine hundred fifteen kilobytes worth of evidence, that is.)

I have two unpleasant things to write about this morning, and I decided to pick the one that’s foremost in my thoughts right this moment. Don’t worry, the other one is much more depressing.

Remember the Quantum Snap! Server? I’ve ranted about it before. I’ll probably rant about it again before it’s gone, too.

It crashed. 10:00am, and it crashed. The same error messages, same symptoms. Three months out of the past four this thing has crashed once per month. I’m tired of it, and I hope I can convince Corporate of 2 things:

1) Karel was wrong about buying the Snap! server, and what we should really do is upgrade the capacity of our existing Netware server, the one that performed so well for us over the years but just ran out of necessary space.

2) Karel, while wrong, isn’t such a freakin’ moron that we should find someone else to do his job. He just made an honest mistake, and that shouldn’t be held against him.

Wish me luck.

Without going into too much detail, my first-ever (yes, Entercom’s a bit behind the times in some ways) official Employee Review went fairly well. That’s one less thing to worry about in my permanent record, I suppose. It has been noted that I need to work on my organizational and communications skills and sometimes my attitude leaves a bit to be desired. All in all, though, it was a positive review. Yay!

The really big news of the day is that our problems with The Beast are all over. You see, it finally dawned on me that I should contact 3Ware about their RAID controller. After a couple of email exchanges, this is what I was told:

“If you have a 5xxx series controller, pressing F9 instead of ENTER will accept dissimilar drives into an array.”

Aaaaaargh! It was that easy all along? How about a hint on the screen that ENTER isn’t the only keypress available? How about something in the printed docs? Huh?

I don’t care too much about documentational and user-interface stupidity right now, though, because The Beast finally has a working 500 gigabyte array onto which Enco files are being copied as I type this. Hot diggity damn. That’s one major problem off my back. Two, if you count the Employee Review…

You may recall the problem we had this past weekend with our Enco servers. While we did get the main server back online with a surprising minimum of fuss, our standby server is still awaiting the correct replacement drive. The main cause of our problem is 3Ware’s insistence on having only 100% identical hard drives in any given array, a requirement quite out of step with most sensible RAID controller manufacturers.

Fine then. Since we don’t want to completely rebuild The Beast as a SCSI-based machine (bloody expensive proposition, that) we figured we’d hunt down another drive of the exact same model number as the others already in the machine. This way we can get a bit more mileage out of our lamebrained investment in 3Ware’s IDE-RAID technology. Great idea, right?

Wrong!

Same model number, different month, slightly different capacity.

It was no surprise that when I tested the new drive anyway (Gary would have insisted on it) I see this:

Delightful. Just delightful. It looks like we’re going to have to spend just shy of $1000 on hard drives to replace the entire bloody damned set. And that means at least two more days before our standby server is once again on standby.

I hate working without a ‘net…