Fun from a datacenter

[bynkii.livejournal.com]

So here I am in day two of sitting in a datacenter in Tennessee.

Why am I here? Because of a dead server mirror. Now, you're all thinking "But John. It's a dead mirror. A retarded monkey can fix a dead mirror. You just slap in a new drive, rebuild it and bob's your uncle"

While this is true, it would appear that there is a retarded monkey convention, and there's none in the entire state of TN. It would also appear that the company they normally get service from has all of their retarded monkeys fully occupied. So, their service provider knows I live in KCMO, which is near TN.

I get a call on Thursday

"John, we need you man"

Details follow. I tell them "I can only do this over the weekend, and my "you don't have a good backup policy, confused RAID for a good backup policy, and now I'm going to fuxx0r my weekend for you" fee WILL be paid.

"No problem, the money they stand to lose is FAR more than that"

"Okay then"

Conference calls happen, hurried teen-watching is arranged, (poorheather fuckin' ROCKS here), and off I go.

Now, I know datacenters require cooling. We all do. But here's one...PUT THE SERVER RACKS OVER THE UNINTERRUPTED ROW OF VENTS instead of MAKING POOR BASTARD IT PEOPLE SIT OVER THEM. Dear GOD, I swear by last night, I wasn't pissing, I was serving the urinal a gelatto. There's a friggin' thermocline at about 4' off the floor. Goes froom 50°F to 75°F in 4". I will say there's no danger of falling asleep. Well, until they hypothermia kicks in. But then you're dead. <Homer>Mmmmm....sweet warm death.</Homer>

After finally getting a good image, (holy shit but six million files is a pita. What kind of idiot has a mission - critical server and no fucking backup? Oh wait, the one that is paying me to fix this. With a backup, three hours on thursday. Without one, well, I probably cost a LITTLE less than a good backup system at this point, but if I were to have been here another day, that would be in doubt. My "fuxx0r my weekend" fee is quite high.)

Oh yes, the unanswered question. Why don't they have a backup system? Because the...moron who programmed the system on this box says "It backs itself up". <sigh> Dear people who are, i'm sure, quite good at their jobs, but really bad at IT...backing up MySQL data and backing up your servers are really VERY DIFFERENT. As I'm sure you're now aware.

So get the backups done, now watching the RAID rebuild. Joy. Because unlike what people think, RAIDs don't rebuild in a second. Oh yes, dear Apple Server Engineers. I love Mac OS X Server. It's a lovely product, really. I use it extensively. However, perhaps, just maybe, you could set your RAID rebuild dialogs to say "Time Remaining: Two Hours and 33 minutes" instead of "Time Remaining: One Hour and 93 minutes". It would make you look a lot less stupid. Really.

Oh crap, I think one of my nipples just froze and fell off. <SIGH> Maybe they have some Nipple Formula Krazy Glue. After all, it can hold a pr0n star in mid-air.

People...spend the money on the backups. I like getting this rate, but it's boring as hell, and it makes me hate you. If it runs your business or your life, BACK THE DAMNED THING UP.

Comments

[mouser.livejournal.com]

I used to stick boxes of paper on stupidly placed vents.

Actually, I was more impressed by the guy that moved his big old tower (in the '80s) under the thermostat. Those things used to REALLY kick out the heat! Then everyone wondered why we were all freezing...

[tecie.livejournal.com]

*nod* One oif my first big assignments when I became a sysadmin was to rebuild the TSM environment. The server died a horrible death and nobody ever made a backup of the TSM server or had the recovery plan file, so it was 100% guess work what the DB looked like to begin with as we scanned a couple of hundred tapes to find the last full copy of the db, since we couldnt figure out what the SAN looked like either.
Now I keep a mksysb DDS4 tape of the server in my desk just in case it randomly keels over one day.

[manuka.livejournal.com]

Nice. Wr0d to KC.

BTW, who do you work for? We occasionally have need to rent mac-type geeks for our video guys, since we IT staff don't hang with the cool mac kids.

[bynkii.livejournal.com]

that rather large Insurance/Financial services company on Broadway, but i've some flexibility, so if you need local support, lemme know.

[manuka.livejournal.com]

Good to know - we usually use the guys down at River City, but they're pretty busy sometimes. One of my backburner projects is to get the XRaid up and running in some sort of usable fashion other than having each half of it used for each of the 2 Final Cut machines. They want to be able to share it, but I think we need to get them to buy XSAN first.

[bynkii.livejournal.com]

yeah, if they want to really share it, Xsan is your friend, and pretty damned cheap too. The fiber hardware can get pricey, but Snakes on a Plane there.

[manuka.livejournal.com]

the fiberchannel hardware we've already got, though - I understand the software is about a grand, which isn't bad at all, especially if apple does nonprofit licensing.

[bynkii.livejournal.com]

non-profit...Maaaaaybe, but doubtful, as they only really do discounts for .edu institutions, but you can always ask your Apple rep.

however, you only really need one or two copies of it, so it's not a hideous expense.