KNRK’s program director called me over so I could look into why his music scheduling program was running so darned slow. He’d already performed the obligatory reboot to no avail, so I dutifully went to his office and watched the software misbehave as claimed. The problem manifested as long pauses of up to 90 seconds when performing random tasks.

I checked the Task Manager. NTVDM took up most of the CPU power during those pauses. Fine, so the Virtual DOS doohicky was working overtime. Why? My next thought was to check the program’s working directory on the server for telltale files. I found them, alright. I found four thousand of them.

VDM(hex-numeral).tmp files. All of them zero bytes in size, ranging in age from a couple of days to a couple of years. Apparently the VDM never, ever deletes these puppies. So I did, and it… almost helped.

I looked deeper. Deeper into the directory structure, that is. In one of the subdirectories I found something that really surprised me: 65,000 of those VDM*.tmp files.

Whoah.

I wasn’t at all surprised that deleting all of those files made the program work better. In fact it ran considerably faster than the PD expected, as he’d grown used to it being fairly slow. My next amazing feat will be the implementation of a timed script that regularly removes those damned “temporary” files…

Say you’re a service vendor, and you’re working with a company to set up additional services. This is a company you’ve been dealing with for a while now. Now say that the IT Department at the company in question sets up, for instance, some port-forwarding rules on their firewall. You’re able to connect to the computers you need, and can install the software. Then you come across a problem, and you think it’s firewall-related. Who do you contact?

Apparently, if you’re our streaming audio provider, you don’t contact the IT Department. You send an email to someone else entirely at the company, at 9:30 in the morning, and wait for someone to get things taken care of.

I found out about the email in question at about 3:00 in the afternoon when the guy who received the email asked, “So, did you fix the firewall problem?” (My reply, of course: “Uh, what firewall problem?”)

This gets even better, as it tends to around our offices. It turns out that while the remote-access port was working well enough as I configured it, the vendor decided to change the port that the encoder listens on… to the outside port I gave them to connect to… not the original standard port that I had the firewall forwarding to. If they’d just left things the hell alone, we might very well have been streaming by day’s end. But no, oh no. Clearly they know more about this stuff than I do. Who am I, anyway? I’m just the IT guy, after all.

I set up port-forwarding on my proxy/firewall machine. Not only did I manage to make it work in fairly short order, but I also managed to do it correctly the first time.

So, the impetus for this crash (if you’ll pardon the term) course in firewall geekery? The powers that be want KKSN and Charlie streaming, immediately if not sooner. We don’t have enough hard IP addresses to do so. Thanks to a suggestion from Corporate IT, I came up with another solution that I wasn’t sure I could pull off, but still looked like a better idea than ordering another bank of IP addresses from our ISP. (I hate changing DNS for a dozen machines and all sorts of domains. Gah.)

Now if only we could get to a point where, you know, I could work on the normal parts of my job that I’ve had to largely ignore for, oh, quite a few months now…

I’ve probably walked past it hundreds of times since it was hung there, but here’s what I found dangling from the upper rungs of one of the telco racks in the server room:

As I was resizing the picture on my computer, I remarked to The Buzz’s program director as he passed by my office, “This should be on every piece of equipment ever made.”

He replied, “That should be on every woman ever made!”

It’s not every day that one catches the 8:07 #9 Broadway bus downtown, twice in the same day.

This odd little musing is brought to you by the crashing of this very webserver, due to my completely failing to learn one particular lesson from the aforementioned “last year’s debacle.”

That’s right! Not only did I arrive at work at 9am, I also arrived there at 9pm. This time it was to set the webserver to boot with a uniprocessor kernel instead of the SMP kernel, which was the previous default, and which the hardware in question is dangerously unstable on. Ugh.

While I was there I fixed a few other things, then came home. Again.

Now, if you’ll excuse me, I’m going to finish my long-delayed dinner and go to sleep. Bah.

Walking in the door, almost no sleep under my belt (for various reasons), I was immediately thrust into a tricky and delicate hard-drive swap operation as part of a swap of studios for Charlie and Kisn (AM). That came off remarkably well, but it did involve three very intense hours of work.

The middle part of my workday wasn’t so bad.

Almost at quitting time for the day, most of everything went dark. Yep, the UPS for the main part of the building went kerblooey. Again. (Long-time readers may be familiar with the troubles that damned piece of equipment has given us. Yes, indeed, I hate it a lot.) Cue a solid hour of running around, desperately trying to get things working again in the shortest possible time.

Luckily, I learned a lot of lessons from last year’s debacle. For one thing, I’ve upgraded a lot of the Linux servers so that services generally start on their own. I also know exactly which systems are troublesome, and how to get them back to life again. However, one new problem reared its head: Our main file-and-print server wouldn’t let most folks sign on until I thought to run a ‘DSREPAIR’ routine. Ugh.

I’ve already accumulated enough stress to last me the entire week. Powers that be, if you’re reading this, please take note. Thank you.