grayhawkfh.livejournal.comSo, some new equipment was installed in our server far (I think it was air handlers). As part of the contract, we were to receive a training session, and it would be taped so it could be used in the future.
Now, a quick note about the power to the server farm is in order. There are two emergency cutoff switches to the room. The external one, if triggered, will shunt power from the emergency backup generators to the farm. The switch inside the farm, however, is "Dead Man's Switch". It kills all power to the servers. Period. We return to the story in progress...
I was not present for the training. After all, it wasn't for me. If there's a problem with the air handlers, I call the necessary maintenance staff. So I can only describe what happened as it was told to me.
But what apparently happened was that the in-duh-vidual giving the presentation reached over to point out the dead man's switch, and learned the hard way that his depth perception was not all it should be.
Next thing we know, the server status monitor is screaming bright blue murder at us. I was coming back from running a ticket to see the monitor glowing red, the lead network engineer standing on his desk (so he can see over the cube farm walls) with a phone in one hand and a stress ball in the other, and the hell desk phones ringing madly off the hook.
This was almost 2 hours ago. The trainers beat a hasty retreat (lest they be tarred & feathered), and we're still working on getting everything back up. I have...6 tickets waiting for me to work on them, and I just got word that I'll be able to work on them. Out opinion is that the trainers need to get their arses back here with a company credit card so we can go set up a very large bar tab somewhere...
Oh, did I mention that this was taped? Did I also mention that the downtime was being tracked? I think that there's a trainer that will be looking for a new job soon.
(x-posted to my personal LJ)