techrecovery | Aug. 7th, 2010

Background: 24-hour Network Operations support for probably the largest, best-known global imagery sourcing company.

Last night I got to call London around 3am Pacific.

London: "London picture desk."
kizayaen: "Hey. This is $KIZ from the NOC in Seattle calling. We're seeing no-poll-response alerts for $LONDON_SWITCH_04 in our Operations Manager console. 05's failing polling too, but hasn't hit the alert threshold yet. How's your connectivity looking over there?"
London: "It's okay as far as I know. Let me ask around and see if anyone's having problems."
kizayaen: "Sure."
London: "... It looks like we're all fine her... no, wait, one of my colleagues seems to be unable to connect... nope, looks like we're all down. Intranet is working but we've got no external access."
kizayaen: "Right. 05's now failed and we're getting poll failures on 01, so going by nomenclature, you've got at least three of five switches down."
London: "Let me check around on some things. We've got workmen here, I'll try and find out if this is related to anything they're doing."
Alarm bells: "AAAAAAAAHHHHHHHHHHHHHHH"

Sure enough, the workmen onsite had accidentally pulled some of the wrong cables in the course of whatever the hell they were doing. Whoopsie.

But man, I loved the subtext of that phone call. It was possibly one of the most fun phone calls I've ever made in my life.

"Hi, London. This is the opposite side of the world calling. Your network has just gone into screaming fits of uncontrollable shits. We know it before you do. You'll be able to verify it independently in... just about... any second now."