Category: Work

  • Timeline Of A Debacle, A Situation Report

    Let’s recap the story so far.

    Almost exactly a year ago, pressed for network storage space, we evaluated two solutions. One involved expanding the existing server’s capacity, the other involved putting our faith in new, unproven technology. Naturally we went with the unproven technology.

    July of last year, we suffered the first crash of the Snap! server. Considering it a fluke, we soldiered on blithely.

    Fall of last year, the monthly crashes began. Stress mounted.

    Two months ago, the decision to go with the original “other” plan is reached, and we order the replacement drives only to find that we can’t upgrade the original server without also destroying the operating system partition in the process, thus losing us several days of productivity.

    A month ago, we came up with the final possible option, replacement of the main server. The clever part was that we’d use the drives that couldn’t be safely used in the old server, thus getting our money’s worth out of ’em.

    Today, the new server showed up.

    Guess what? We’re still not out of the woods. You see, we can’t use the drives we purchased for the other server. Turns out, upon investigation, that you have to buy special “Hot Plug” drives for this kind of server. Lovely. That’s just effing wonderful.

    My greatest fear right now is two-fold: One, that someone is going to wake up and realize what a mess I’ve caused and boot my ass out the door; two, that it’s not over yet, and even after we get the new drives in and the machine is running, something else will go horribly awry.

    It’s a gift I have, this knack for calamity. Too bad I haven’t figured out how to turn my talent for disaster to my actual benefit, eh?

  • Monday, Much Like Friday

    Bad news: Just like on Friday, the Snap! server crashed this morning, costing me an entire productive day of work. The symptoms were identical to Friday’s, which were atypical for the monthly crashes we’ve grown used to.

    Of course, when I say “grown used to” I mean “grown increasingly fearful of each and every month, to the point where jaws clench and stomach acids churn and tempers fray like cheap twine.” Or something to that effect.

    Good news: We’ve finally ordered the parts for what will become the new central office server, replacing Tokimi (the Snap! server was named after the “evil” goddess from the Tenchi Muyo OAVs, yes) and Arthur (our current main Netware server, named according to Ben’s “Holy Grail” theme). It’ll only take me three solid days and nights of work to make the transition… if nothing goes wrong.

    Starting either tomorrow or Wednesday I’ll probably be living at the office for a few days. Anybody have a cot I can borrow?

  • Neverending Workday

    I’ve had a dark, unhappy day at the office, and it’s not over yet. Left for the evening are the tasks of upgrading RAM in the traffic department’s workstations and upgrading Tapscan (sales software for radio). Both tasks have to wait until the respective departments are fully done for the day, so… for the moment I have time to write.

    I won’t bore you with the details of the day. Suffice to say that I had more excitement with Corporate about the new server we need, there were lots of weird little fires of which about half were manageable, and I have at least one full workstation rebuild in my immediate future. And how lucky am I that one of the dead PCs is a KNRK studio workstation? Yep, I’m just that lucky.

    Oh, did I mention that because of this evening’s oh-so-necessary jobs, I have to miss the planned Wednesday-night viewing of Cowboy Bebop and blow off my family, who were planning to see a baseball game tonight? I could only do one or the other, but now I can’t do either.

    If you need me, I’ll be in my office.

  • Gracelessly Admitting Defeat

    Let’s skip the “ado” and go right to the email I sent to my Corporate Boss a few minutes ago:

    The story so far: Last summer, strapped for network storage space on the main office network, our hero researched alternatives and chose a Network Attached Storage device (the Snap! server) over attempting to upgrade the second RAID array in his existing Netware server. This solution cost about $2500.

    Late last year problems arose; the Snap! server began a monthly cycle of crashes, costing the office a day of downtime on every occurrence. This became increasingly unacceptable, and Quantum’s tech support was lacking. (“We have no idea what’s wrong with your machine. We’d send you a replacement, but we don’t have any.”)

    After one crash too many, more accurately two crashes on the same day, our hero swallowed his pride and admitted his mistake in choosing the Snap! appliance instead of expanding the Dell Poweredge server’s storage capacity. Hard drives were ordered, to the tune of about $1600.

    Then the other shoe dropped. It is impossible, according to Dell, to delete only one array from a Poweredge 2300. One must remove all arrays, then recreate. This would mean fully reinstalling Netware onto the core office network server. All of a sudden, the “upgrading the Netware array” solution has become utterly unworkable.

    The problem as it stands: Live with a day of downtime every month, in addition to the other problems the Snap! appliance is causing? Spend somewhere between two and five days attempting to recreate the Netware environment we currently enjoy, with no guarantee of completion date or even success of any kind? Drop another big bundle of cash on Some Other Solution?

    I’m at a loss, and so I’m punting this up to the Powers That Be at the home office. I look forward* to discussing this at some very near date.

    Yours, etc.

    ( * Okay, that’s a lie. I don’t look forward to anything involved in this giant ugly snafu anymore. I’m ashamed and disheartened at the whole damned affair. But we need a solution, so I’ll carry on until we have one.)

    There are days when just being me is enough to wear me out. This is one of them.

  • Random Post-Weekend Entry Du-jour Thingie

    First, the weekend. I didn’t do anything. No, really. On Saturday I started and finished a scenario in Heroes of Might and Magic IV. Unless you count making a top high score as an accomplishment, I accomplished nothing Saturday.

    Sunday wasn’t much better. In the morning I started a brand new Assassin in Diablo II. By nightfall she was a level 21 in Act III and had points stored up in anticipation of hitting level 24 so she could start dumping points into her two main traps. Yes, I’m trying out a Trapper build. I’m astonished at how effective traps can be, especially compared to the cumbersome and somewhat buggy Martial Arts skills. (“What do you mean, I right-clicked on the monster and nothing happened… again? Argh! Stupid charge-release skills!”)

    And then it was Monday, and while two days of hardcore gaming hadn’t entirely cured the anxieties of eleven straight days of work, it had numbed me to the point where I didn’t resist heading to the office. This is probably a good thing. I didn’t accomplish anything at work today, either, except resolve two stupid email-related problems.

    You see, Lancelot has been replaced by Mass. (Yes, the box I had so much trouble with recently is now almost fully operational.) Today I intended to install Courier IMAP. I followed the instructions, installed the software, ran the start-up command as instructed… and could not log in from any machine at all. Argh!

    Almost five hours of starting, restarting, reinstalling, poking, prodding and cussing finally led me to try the other startup script provided. Voila! Gee, it was so effing easy…

    But wait, there’s more! After telling all of the other servers to start sending their little alerts and what-not to Mass, I realized that Mass was incapable of receiving email. “Huh?” It had worked before, and at first I was convinced that the firewall was screwing things up. No, of course not. See, had I thought at first to look at Qmail’s logs to see if it was detecting connection attempts, I’d have known that I’d broken one of the configuration files shortly after bringing email online. For the want of an ‘=’, the server was lost…

    And that, folks, is how a guy like me blows an perfectly good eight-hour workday straight to hell. Sad, wot?

    To highlight my utter sadness quotient, I was prodded into taking a 500-question purity test. (Look at Lil’s current NOTD; I’m too lazy to copy and paste linkage tonight.) My dear friends all scored in what we shall call the “lower half” of the percentile range. Not me, though.

    I’m 69% pure.

    On the one hand, “sixty-nine, dude!” On the other, how can my friends stand to be around someone so relatively square? It must be the power of contrast. That, or my stunning charm and personality.

    No, that wasn’t meant as a joke, so you can stop laughing now. Thanks.

    And now for more gaming fun. Alex and I spent a very enjoyable hour or two this evening playing Team Holomatch in Star Trek Voyager: Elite Force. He and I put on our Johnny Bravo models and went around killing Gauron, Janeway, Seven of Nine and The Tick. Repeatedly. Much laughter ensued. Think of it as a father/son bonding experience. “Woo! You totally smacked that little fairy, didn’t you! Ah, that’s my boy. Hah! Did you see that shot? Damn, I’m good.”

    Anyway, tomorrow evening I’m to spend the evening with Lilith, Geoffrey and whoever else happens to be at their place. I’m looking forward to it, oh yes indeed. Talk about incentive for peeling myself out of bed in the morning!

    And that’s just about enough out of me for tonight. It’s amazing that I approached this entry expecting to only produce three or four paragraphs. This proves once again that I’m entirely too enamored of my own cleverness and the sound of my voice. Then again, if you weren’t just a bit impressed by it, you wouldn’t be reading this. If you don’t mind, I’ll go to bed smug now…

  • Think of it as a strongly worded “unsubscribe” message.

    I’ve ranted about the Quantum Snap! Server NAS appliance before, with a certain regularity given its tendency to vapor-lock once per month. It happened again today, but this time with a twist. Shortly after the final rebuild and the return of logged-in users, it crashed again.

    But wait, there’s more! As I fumed about the utterly incomprehensible behaviour of the machine, what should appear in my office email but this:

    Special Offer on 960GB Snap Server 12000!

    Dear Karel,

    As our way of thanking you for being a loyal Snap ServerÆ customer, we would like to extend you a special offer. Purchase a 960GB Snap Server 12000 between now and May 30, 2003 for only $8,499 – a $3,500 savings over MSRP!* At less than a penny per megabyte, use a Snap Server 12000 to solve common storage problems.

    960GB Snap Server 12000

    Now Only $8,499! You Save $3,500!

    What incredible timing! It’s as if they knew I really needed a replacement unit! Here, therefore, is my joyful reply:

    Dear Quantum, or Snap Appliance, or whoever you are now,

    Your sales missive could not have arrived at a more appropriate time.

    If your “Snap Appliance” product hadn’t crashed on us twice in a row today, I might think about spending almost five figures to get an even bigger, better version. Sadly, the Snap! 4100 we bought almost a year ago decided that crashing mysteriously once per month isn’t enough and has graduated to higher levels of mayhem.

    Yes, I’ve spoken to tech support almost every time this happened. They don’t know what to make of it either. They offered to send a replacement, but there weren’t any in stock last month. Gee, what a shame. Sucks to be us, hey?

    Tell you what. Send me one of these $13k beasties ‘gratis’ and I’ll think about not badmouthing you to every peer and colleague I know. (Did I mention I work in the broadcast industry?) I’ll be sitting here watching my drive array rebuild again, holding my breath. You betcha. Do the words “you’ve lost my business, stop trying to get more” mean anything to you?

    Good luck, I get the feeling you’re going to need it. Consider this my idea of an “unsubscribe” message.

    I believe this is an exception to the adage about composing email while angry. What do you think?