ext_121382 ([identity profile] valiskeogh.livejournal.com) wrote in [community profile] techrecovery2006-09-20 09:19 am

D'oh! we must alert all of IT!

my coworker and i discovered that email is down. i called and alerted the helpdesk that they may start receiving calls and that we know about the problem while he called the networking team to check to see if it was a networking issue before calling the email guy and letting him know (he was at HOME, probably asleep... typical...)

Sandra from the helpdesk walks over to check on the problem. she says she needs to alert all of the IT department if Email is down as well as create a work order. we tell her that yes, it's a server problem, we aren't sure of the nature yet (from webmail and even outlook or thunderbird, to the layman, it doesn't seem like there is any problems, but email is NOT getting anywhere, not sending and not receiving) she says okay, and she's going to go alert all of IT.

5 minutes later something hits me. i turn to coworker ande say "wait... she's not doing what i THINK she's doing is she?!?!" we both start cracking up.

i walk over and sure enough...
She's sitting there trying to send an ALL IT Email alerting everyone that Email is down.

Valis

[identity profile] major-error.livejournal.com 2006-09-20 02:33 pm (UTC)(link)
In the ever-funny words of Bill Engvall (http://en.wikipedia.org/wiki/Bill_Engvall): "Here's her sign..."

terribly OT

[identity profile] thalionar.livejournal.com 2006-09-20 06:06 pm (UTC)(link)
Did you know that Bill Engvall is part owner of a bull in the PBR, named "here's your sign"?

Very pretty bull, and I laughed hysterically when I saw it on Sunday.

[identity profile] ladynisa.livejournal.com 2006-09-20 03:20 pm (UTC)(link)
Dude... way too early for that level of t3h stoopid. O.o

[identity profile] hisamishness.livejournal.com 2006-09-20 05:08 pm (UTC)(link)
Tell her to drop a quick email over to the admins to let them know she can't alert all of IT via email.....

... or ask her if she can pick you up a spare can of squelch while she walks around to notify people...

[identity profile] knottie.livejournal.com 2006-09-20 05:35 pm (UTC)(link)
At my last job, when something similar happened, the IT director had a much better idea--mass voicemail distro to everyone (1200+ people). And then his message got cut off because he took too long, so he resent it. Took down the whole vmail system under the load.

Someone finally taped notices to all the doors. Technology: 0 Scotch Tape: 1

Whenever I reboot the wrong server, I console myself that it's not THAT bad. ;)

[identity profile] redqueenmeg.livejournal.com 2006-09-20 07:26 pm (UTC)(link)
Yep, we'd always get calls when email was down demanding to know why we had not emailed everyone that email was down...

[identity profile] the-s-guy.livejournal.com 2006-09-21 01:18 am (UTC)(link)
Sounds like an opportunity to forward it to the relevant managers with the added text "Please explain to your subordinate why this would not work."

[identity profile] redqueenmeg.livejournal.com 2006-09-21 12:44 pm (UTC)(link)
Ah, and that would be a one-way ticket to Attitude Adjustment City, with stops at Permanent Writeups In Your File and Lecture On Customer Service From Your Manager.

At least at that place of work.

[identity profile] the-s-guy.livejournal.com 2006-09-22 02:03 pm (UTC)(link)
If it was anything like the place I used to work at, it stopped there as well. I think they had to build another warehouse to hold my personnel file in the end, and I nearly had my own chair in the manager's office, I was pulled in there so often.

New bosses usually came out of such encounters looking a little ragged around the edges, having extracted precisely no promise from me that I would not do exactly the same thing in future.

[identity profile] redqueenmeg.livejournal.com 2006-09-22 02:07 pm (UTC)(link)
Yeah, that's the difference between being replaceable and irreplaceable. We were all the former.

[identity profile] the-s-guy.livejournal.com 2006-09-23 02:32 pm (UTC)(link)
Heh :) I was never irreplaceable, merely worked for people who spewed so much red tape that they couldn't make a move without pulling down half the empire.

[identity profile] network-nerd.livejournal.com 2006-09-22 12:31 am (UTC)(link)
We did! Honest!
azurelunatic: Vivid pink Alaskan wild rose. (Default)

[personal profile] azurelunatic 2006-09-24 09:49 pm (UTC)(link)
Sounds like end-users have a hard time distinguishing between "planned outage" and "unexpected problems".

I suppose "You will get the e-mail notifying you about the outage once e-mail service has been restored" would be another writeup moment?

[identity profile] redqueenmeg.livejournal.com 2006-09-26 02:18 pm (UTC)(link)
Possibly, although that certainly wouldn't happen until I could have gotten the user to understand what I was saying, and I can assure you, saying that quote just once to them wouldn't have done it! :)