After an hour or so of gaming followed by most of an hour on the phone with a particular someone special, I decided to indulge in a pasttime I’ve all but given up in the last few years. Yes, I put on my headphones and listened to music for a while. Hey, the computer was on anyway, trying valiantly to download some anime for my later enjoyment.

(Keep in mind, now, that a week ago we upgraded the memory in this webserver, and that it hadn’t had a crash for roughly a week before that, even, thanks in part to some configuration tweaks I made a week or so prior to the scheduled upgrade.)

Imagine my chagrin when I decided to check my email one last time before bed (what? oh, like you don’t. go on, pull the other one.) only to stare in horror at the hourglass cursor as Thunderbird utterly failed to connect to my server. I switched to the console screen and tried a ping. No dice. I could ping, and visit, any number of other addresses… just not mine. I took the next logical step, of course. I called Infinity Internet‘s tech support number and left a detailed message. Once I completed the call, on a whim born out of what could be called a stubborn refusal to accept facts, I tried the ping again.

You can probably guess what happened.

Once I finished rolling my eyes in disgust, I placed another call to Infinity’s support line to let them know not to bother rebooting my server…

As near as I can tell, something external to the server went dead for a little while. The evidence is this line in the /var/log/messages file:

e100: eth0: e100_watchdog: link up, 10Mbps, half-duplex

The gist is, that notation tells me that the network cable was unplugged or inactive in some fashion, then came back online, which spawned that “watchdog” message. (Now if only there were watchdog messages about the link being down, I could get a handle on how long we were offline. Argh.) And since that’s the only indication that anything at all went wrong, I’m going with the “someone tripped over a Cat5 cable” theory, until a better theory comes along.

All’s well that ends well, though.

An update on my massive webserver-status post below: The daily random email alias generator is running again, for those of us who made use of it. (Oddly enough, that generator is the last remnant of code from Monaural Jerk to be used anywhere on our server.) Also, I found the problem with emailed comments showing up with empty “From” addresses. Turns out that it’s a problem related to the installation of Spam Karma 2, and I’ve tweaked the code in that plugin accordingly. (It tries, apparently, to make the “From” address something like “wordpress at domain” but it wasn’t working. I simply changed the parts of the code that created the “Reply-To” header and turned them into “From”.) Now any email spam filters (such as SpamAssassin) won’t be scoring “down” comment notifications because of a missing rather-important email header.

Not bad for an hour’s work, eh? Speaking of work… I need to go continue looking for that.

We moved to this new server a few weeks ago, and as of Monday we’re running on 256 MB of RAM. Once I felt comfortable that the system is stable enough after the upgrade, I started fixing things, but with the job hunt and other life stuff (and the occasional heroism & villainy), I haven’t gotten around to everything that needs doing. Here’s where things stand so far:

Yay! The RAM upgrade seems to be working precisely as expected. We’re no longer chugging periodically, nor have we crashed the server completely in well over a week. Between the memory tuning tweaks to SpamAssassin, MySQL and Apache I think we’re good to go for the long haul.

Boo! I lost my gallery. It’s entirely my own fault, mind you. When I upgraded to Gallery 2, back on the old server, I failed to add the new directory structure (most of G2’s files live outside the web path) to the backup routine. I think I managed to salvage the images themselves, but the install is completely hosed. If I want a gallery, I’m looking at a completely fresh install and a fair bit of tedious re-posting of every image. Ouch.

Yay! On the other hand, everyone else’s gallery is once again fully functional, now that Jhead, ImageMagick and NetPBM are installed on the new server. Aren’t you ladies glad that I never got around to pushing for G2 upgrades?

Boo! With the loss of my gallery I’m looking at the prospect of re-coding all of those image links within a dozen Mai Otome recap postings in the anime site. I’m not convinced that this is a good use of my time, to say the least, considering that not only will I have to re-code the links (Gallery 2 does… interesting things with image URLs) but re-upload a buttload of recaps to the new gallery. Ugh! So… if you’re wondering where the hell the anime site went? The answer is, “Away.” I have the database dump file (I was backing that up at least) so I can extract the series reviews, which I consider to be the only content really worth saving. My current idea is to re-post those here under an Anime “Reviews” category right alongside those other Reviews categories I never use. Er, yeah.

Yay! Yesterday I set up the firewall code designed to mitigate the effect of brute-force SSH login attacks. It’s a more elegant solution than the Swatch-based system I implemented back at the job site, and anything that does the job without adding to the server’s process overhead makes me happy.

Boo! I haven’t put the backup scripts into place yet. Uh, yeah, I’ll be getting on that this weekend. Part of the problem lies with the fact that unlike my previous server, this one lacks a failover machine that I can simply mirror the files and store database dumps on. If I want to back this thing up properly, I need to either manually download database files and website archives on a regular basis or find something that’ll do so automatically, which is far, far more appealing both from a “reduction of tedium” perspective and a “make sure it actually gets done” perspective. Believe me, folks, I’m working on this. Every day that the hard drive doesn’t die in this server is a blessed gift from the goddesses.

I think that covers the highlights. Overall, I’m pleased to be in complete control of my own webserver at long last. (Not that the previous server was out of my control, I suppose, but there were… limitations.) I’m also delighted by the available bandwidth, as we’re no longer competing with six outgoing audio streams and a bajillion “now playing” text file requests for the resources of a mere single T1 connection. Now I just need to clear a few more hurdles to get back to nearly where we were before, and then I can (gasp!) try some new things.

When I get around to them, that is.