Never Press The Big Red Button

[grayhawkfh.livejournal.com]

So, some new equipment was installed in our server far (I think it was air handlers). As part of the contract, we were to receive a training session, and it would be taped so it could be used in the future.

Now, a quick note about the power to the server farm is in order. There are two emergency cutoff switches to the room. The external one, if triggered, will shunt power from the emergency backup generators to the farm. The switch inside the farm, however, is "Dead Man's Switch". It kills all power to the servers. Period. We return to the story in progress...

I was not present for the training. After all, it wasn't for me. If there's a problem with the air handlers, I call the necessary maintenance staff. So I can only describe what happened as it was told to me.

But what apparently happened was that the in-duh-vidual giving the presentation reached over to point out the dead man's switch, and learned the hard way that his depth perception was not all it should be.

Next thing we know, the server status monitor is screaming bright blue murder at us. I was coming back from running a ticket to see the monitor glowing red, the lead network engineer standing on his desk (so he can see over the cube farm walls) with a phone in one hand and a stress ball in the other, and the hell desk phones ringing madly off the hook.

This was almost 2 hours ago. The trainers beat a hasty retreat (lest they be tarred & feathered), and we're still working on getting everything back up. I have...6 tickets waiting for me to work on them, and I just got word that I'll be able to work on them. Out opinion is that the trainers need to get their arses back here with a company credit card so we can go set up a very large bar tab somewhere...

Oh, did I mention that this was taped? Did I also mention that the downtime was being tracked? I think that there's a trainer that will be looking for a new job soon.

(x-posted to my personal LJ)

Comments

[liakela.livejournal.com]

I see.... the title of that instructional video....

"What Not To Do: An Idiot's Tale"

Heheheheheh. Sucks that you're all in "omg!" mode, but.. funny nonetheless.

[jecook]

LOL.

reminds me of the copious "Emergency exit" buttons all around one of the data centers I had access to during my tenure at [ISP].

We were waiting for someone to show up so we could open the exterior door to get some large equipment out of the storage room, and I innocently pressed one, thinking it was an "unlock door" button like what we had at one of the other sites.

BIG, BIG mistake. It did not cut power, but it certainly gave the security people a *lot* of exercise, as their monitors showed it, and they thought there was a fire or something.

Oddly enough, they still let me into the building after that incident. by myself even.

[tecie.livejournal.com]

LMAO -- we actually put plastic guards over all the EPO switches (you lift the guard and push the red button ... they're made for fire alarms)

[jecook]

There is a name for the guards - they are called Molly Guards. Well, at least in the hackish sense, anyhow.

[loosechanj.livejournal.com]

It kills all power to the servers. Period.

Ooooh, I think I see where this is going already.

[meijhen.livejournal.com]

LOL....that happened a time or two when I was working at .

One time, it was actually valid, as the raised floor was about to flood. This, of course, is what the thing is intended for!

Another time was an accident.....and the last was intentional.

And if your company higher-ups are anything like those at my old company......trainer will probably get a smack on the hand and that's it :o(

[toxico.livejournal.com]

I DEMAND humbly request mpegs.

[bamatone.livejournal.com]

Seconded.

[grayhawkfh.livejournal.com]

If I can get them, I assure you, they'll be posted somewhere...

[chaobell]

As soon as you mentioned the dead man's switch, I just went "oh no."

I seem to recall that the Great Livejournal Outage a while back was caused by some similar incident. :)

[jecook]

It was:

http://www.livejournal.com/community/lj_dev/670215.html

[wyldthyng.livejournal.com]

um.

why bother having the button at all if it's not to be used, EVAR?!?!?!oNEone1one

(not a techie in THAT field)

[wyldthyng.livejournal.com]

nevermind!

the above link re: LJ Power Outage explains it :)

[swwinchester.livejournal.com]

Well, that's the thing. You don't want to NEED to use it, but it exists for several valid reasons. See the comment above about flooding in the raised flooring for a very good example.

I still think that the Dead Man's Switch is an apt name, as if you hit it without sufficient cause, you are likely to be a dead man.

But yeah. Everywhere I've been, those switches have had (at least) ONE "You must flip this cover before you can hit the switch" cover over them.

[trixtah.livejournal.com]

Yes, the place I worked at with the Big Red Switch had SPRINKLERS as fire control in the server room. And only one exit. First sniff of smoke and I was going to hit that sucker...

[wyldthyng.livejournal.com]

yeah, I can see the one at my work in my mind's eye. It's not easily accessible - I mean without going to the trouble of lifting the cover and smacking the button - by which time you'd probably have 6 people tackling you, since it's a call centre ;)

[trixtah.livejournal.com]

AHAHAHA! That's awesome. VIDEO!

I wish they had that for the last place I worked at. For your non-Anzac types, New Zealand and Australia have a bit of a sibling-rivalry thing going on. Just imagine how marvellous it was when the new Aussie guy was standing in the server room waiting for some job to finish, and he saw the big red button on the wall and thought "I wonder what that does?"

Being the empirical Aussie battler, the "suck it and see" kinda bloke, of COURSE he pushed it. Imagine his delight when the whole server room power supply went POOOOUUUMmmm..., all the UPS alarms went off, and the rest of the systems team, including the manager, charged down to see what the problem was.

Unfortunately, I wasn't working there at the time, but they were still telling that story months later. As you would.

[spacebird.livejournal.com]

completely offtopic. But I simply love that icon... it gets better every time I see it. Mind if I yoink with credit?

[geekgrrl-ca.livejournal.com]

I am reminded of the time a bell tech dropped a screwdriver or wrench in one of the major telecomms stations in toronto pre-y2k. It affected comms for allot of canada and parts of the morth eastern usa. I can't remember if the tech lived (it resulted in a nasty electrical fire).

[vvalkyri.livejournal.com]

I've been on the phone with Comcast tech support and had her laughing just as much as I with this. Dear Ghod. (It takes a hell of a long time for a 1997 Thinkpad running Winnt to boot up. And it would be nice if they could see my modem, but other than that...)