Where did my server go...

[jon787.livejournal.com]

So some guys I used to work with lost a server today. It disappeared from the network at around 2pm and they can't physically locate it to fix it.


Proper credit where credit is due:
adam|beer is adamjaskie
Dark-Fx is dark_fx
pyro|lappy is mtupyro
techieb0y is techieb0y
NUXI is jon787 (me)
Not sure if anyone else who shows up is on LJ

13:50 -!- adam|beer [$USER@rock.$DEPARTMENT.$UNIVERSITY.edu] has quit [Ping timeout]
13:50 -!- Dark-Fx|rock [$USER@rock.$DEPARTMENT.$UNIVERSITY.edu] has quit [Ping timeout]
14:02 < Dark-Fx> Who killed rock
15:12 < pyro|lappy> lol
15:12 < pyro|lappy> $FEARLESS_LEADER physically cant find rock
15:12 < MotS> isn't it in the fishbowl?
15:13 < Dark-Fx> WE HAVE NO IDEA WHERE ROCK IS!
15:13 < Dark-Fx> heh
15:13 < pyro|lappy> lmao
15:13 < pyro|lappy> and no cell number for $UNIX_SYSADMIN
15:16 < Dark-Fx> There *is* a machine missing from where rock is *supposed* to be
15:17 < MotS> so there *isn't* a machine where rock is supposed to be?
15:18 < Dark-Fx> MotS: I have a feeling rock is now one of the unlabeled tower machines in the chem basement
15:18 < techieb0y> ...rock is missing?
15:18 < NUXI> pyro|lappy: NNN-NNNN
15:32 < pyro|lappy> NUXI: that's his home number
15:35 < NUXI> he probably doesn't have a cell
15:41 < MotS> dang
15:41 < Dark-Fx> Rock might be in the NOC
15:42 < MotS> Noc your rock?
15:42 < MotS> kinky?
15:57 < Dark-Fx> I think I found rock
15:57 < Dark-Fx> Rock and mandan have bad power suplies
15:58 < Dark-Fx> Rock is the old Askel
15:58 < Dark-Fx> Which isn't askel anymore, I can't ping it at least.
16:00 < Dark-Fx> Oh god that's hoooooooorible
16:01 < pyro|lappy> what room are you physically in?
16:01 < Dark-Fx> $DAS_BUNKER?
16:01 < pyro|lappy> MEEM?
16:01 < HuskyBot> i guess MEEM is really boring...
16:01 < Dark-Fx> No, chem
16:02 < Dark-Fx> Rocks PSU fan is making the worst noises ever.
16:02 < pyro|lappy> it can make a sound that sounds like brittney spears singing?
16:10 < pyro|lappy> Dark-Fx: why isnt rock up yet
16:11 < Dark-Fx> It's brokzored
16:11 < pyro|lappy> $USER_SUPPORT_MANAGER: "that's not a good sign"
16:11 < Dark-Fx> Okay, this isn't rock -$FEARLESS_LEADER
16:12 * Dark-Fx heards to noc
16:12 < pyro|lappy> "uh oh"
16:18 < Dark-Fx> lol where the hell is rock
16:19 < Dark-Fx> I spies a ruckus!
16:19 < techieb0y> ...how does one /lose/ a 4U box?
16:19 < Dark-Fx> We're not sure if it's a 1U or 4U
16:19 < techieb0y> didn't rock get transitioned to one of the V440s?
16:20 < Dark-Fx> I thought so
16:20 < techieb0y> and, do you know rock's MAC?
16:23 < Dark-Fx> not the current one :(
16:24 -!- agmlego [agmlego@$HOST.resnet.$UNIVERSITY.edu]] has joined #UNIVERSITY
16:25 < agmlego|frondeur> So, we're missing a server?
16:27 < Dark-Fx> completely have no idea where it's at.
16:28 * Dark-Fx talks with $COMPUTER_SECURITY_GUY
16:36 < Dark-Fx> back in chem
16:36 < Dark-Fx> Okay, now we think the machine is what arnheim used to be...
16:44 < Dark-Fx> When's the last time anyone's run uname -a on rock?
16:44 < Dark-Fx> Can you tell me what the machine was?
16:46 < agmlego|frondeur> So, what's the deal with rock? It's down, and you're trying to find it to fix it, or what?
16:47 < techieb0y> yeah, it's not responding to ssh, and somehow they seem to not know where it is.
16:47 * techieb0y isn't sure how that's possible
16:48 < agmlego|frondeur> techieb0y: Well, there are a lot of computers on campus...
16:48 < pyro|lappy> Dark-Fx: $SUPPORT_MONKEY_1 says that DNS resolves rock.me, so just start at the network cable and follow the packets
16:48 < pyro|lappy> =P
16:48 < agmlego|frondeur> ...one could probably just hide in a corner somewhere and pout without anyone noticing it.
16:48 < techieb0y> yeah, but servers don't tend to move often.
16:48 < techieb0y> so you'd think you're remember where you moved it to.
16:49 < Dark-Fx> pyro|lappy: DNS stuff is all unlocked for the servers though
16:49 < agmlego|frondeur> Maybe it came alive?
16:49 < Dark-Fx> techieb0y: this is $UNIX_SYSADMIN's failure
16:49 < pyro|lappy> maybe rock got bored and got up and left
16:49 < techieb0y> heh
16:49 < agmlego|frondeur> That's what I'm saying. Pulled a Johnny 5...
16:49 < techieb0y> the concensus is that it's not sn $DAS_BUNKER?
16:50 < agmlego|frondeur> "Johnny 5 is alive!"
16:50 < Dark-Fx> techieb0y: it might be
16:50 < techieb0y> $DAS_BUNKER isn't that big, how can you lose a machine in there?
16:50 < techieb0y> (and if it isn't in there, where else would it be?)
16:50 < Dark-Fx> techieb0y: NOC, $FEARLESS_LEADER mentioned something about REL
16:52 < techieb0y> why'd stuff get moved there?
16:52 -!- pyro|lappy [pyro@$HOST.rovernet.$UNIVERSITY.edu] has quit [Quit: Ex-Chat]
17:11 < Dark-Fx> still no rock!
17:11 < Dark-Fx> cataloged most of the machines in here
17:19 < Dark-Fx> Awesome. still. no. rock.
17:20 < agmlego|frondeur> Rock, why have you forsaken us?
17:20 < Byan> what happened to rock?
17:20 < agmlego|frondeur> It's missing.
17:22 < Dark-Fx> I think it's in the NOC.
17:22 < Dark-Fx> I don't have swipe :(
17:22 < agmlego|frondeur> What about the guy you have with you, or isn't there one?
17:22 < Dark-Fx> $FEARLESS_LEADER does
17:23 < Dark-Fx> We booted the 2 machines I suspected it was, but we might not have waited long enough for it to become pingable
17:26 < Dark-Fx> Anyone up here HAM?
17:26 < Dark-Fx> call up $UNIX_SYSADMIN_CALLSIGN and ask him where rock.$DEPARTMENT.$UNIVERSITY.edu physically resides.
17:42 < Dark-Fx> aragirn: you around?
17:43 < john> I stolz rock
17:43 < john> har har har
21:46 < aragirn> Dark-Fx, what's up?
21:56 < Dark-Fx> aragirn: I was going to ask you to go turn on the machine in the NOC labeled "allouez"
21:57 < aragirn> oh, sorry, I've been off work since 4pm
21:57 < Dark-Fx> yeah, not a problem
21:57 < Dark-Fx> Well, not MY problem.
21:58 < techieb0y> rock still missing?
21:58 < Dark-Fx> Yep!
21:58 < aragirn> how the hell do you lose rock?
21:58 < Dark-Fx> hahahaha
21:58 < techieb0y> wow.
21:58 < Dark-Fx> Because the machine isn't labeled as such
21:58 < Dark-Fx> We have 3 allouez.
21:58 < Dark-Fx> 2 are in the NOC, one is -spare
21:58 < techieb0y> but you think you know where it is, it's just off? Or are you still unsure where it is, physically?
21:59 < Dark-Fx> techieb0y: I'm pretty sure it's one of the allouez in the NOC.
21:59 < techieb0y> ...why are there more than one?
21:59 < Dark-Fx> I think we didn't wait long enough for the machine to fsck before it started to respond to pings.
21:59 < Dark-Fx> One is allouez, the other is allouez-spare.
21:59 < Dark-Fx> Neither of which are the real allouez.
21:59 < techieb0y> lol
22:00 * techieb0y recomends you buy a labelmaker and a putty knife.
22:00 < aragirn> Dark-Fx, and they were labeled rock and rock-spare when they were brought in
22:00 < Dark-Fx> really?
22:00 < Dark-Fx> On the tops of them or something?
22:00 < aragirn> no, that's what $UNIX_SYSADMIN told us they were
22:00 * Dark-Fx has no idea what $UNIX_SYSADMIN was thinking
22:00 < NUXI> Dark-Fx: job security
22:00 < aragirn> then we got a call, "Those are actually allouez and allouez-spare"
22:01 < Dark-Fx> No... Allouez is still in $DAS_BUNKER in checm
22:01 < Dark-Fx> chem
22:01 < MotS> wacky.
22:01 < Dark-Fx> methinks $UNIX_SYSADMIN was confused.
22:01 < Dark-Fx> Or we /really/ don't have a Rock.
22:01 < aragirn> lol
22:02 < NUXI> rock was stolen by underwear gnomes
22:02 * techieb0y still boggles at the idea of loosing a machine in a data center that small
22:03 < aragirn> techieb0y, it was never brought into our data center
22:03 < techieb0y> exactly.
22:03 < techieb0y> $DAS_BUNKER isn't big enough to loose anything in.
22:04 < NUXI> techieb0y: you obviously have never heard of the netware server that got sealed into a wall
22:04 < techieb0y> that I've heard of, yes.
22:04 < NUXI> http://www.techweb.com/wire/story/TWB20010409S0012
22:04 < NUXI> for those that haven't heard it
22:06 < Dark-Fx> techieb0y: /none/ of the machines in $DAS_BUNKER are rock.
22:06 < Dark-Fx> That much we know.
22:06 < techieb0y> ...then where is it?
22:07 < Dark-Fx> Missing...
22:07 < Dark-Fx> There's a spot where cables for rock used to go, at least according to the telcom info.
22:07 < techieb0y> when did it disapear?
22:07 * NUXI looks at his pretty sunfire v100
22:07 < Dark-Fx> It wasn't there.
22:07 < NUXI> its shiny
22:07 < Dark-Fx> I timed out at 13:50
22:08 < Dark-Fx> NUXI: when'd you get a v100?
22:08 < NUXI> Dark-Fx: :P
22:08 < Pyro> maybe that's why rock is down
22:08 < Pyro> someone stole it
22:08 < NUXI> rock was a v100 last I checked 4 years ago
22:08 < NUXI> not idea what model it is now
22:08 < techieb0y> rock was a v120 I thought?
22:08 < techieb0y> then it was migrated to a V440
22:08 < Dark-Fx> Should still be a 1U
22:08 < NUXI> i thought it was a v100 like herman was
22:09 < Dark-Fx> techieb0y: I think it was momentarily a V440 because $UNIX_SYSADMIN brought it up as the wrong box.
22:09 < techieb0y> oh
22:09 < Dark-Fx> I at least have IRC logs of it last being that Feb of this year.
22:09 < Dark-Fx> Only time I could find though
22:09 * MotS cackles.
22:10 < Dark-Fx> The only V440s we had were machines that reported to be what was labeled.
22:10 < techieb0y> and you've accountd for all 4 of those?
22:10 < Dark-Fx> yeah
22:10 < techieb0y> hmm
22:10 < Dark-Fx> Even though askel isn't used at all anymore.
22:10 < techieb0y> heh
22:11 < techieb0y> that's a pretty powerful doorstop, then.
22:11 < Dark-Fx> $FEARLESS_LEADER said it was end of life.
22:11 < Dark-Fx> One of the PSUs in it is dead
22:11 < techieb0y> oh
22:11 < Dark-Fx> One of the PSUs in Mandan are dead too though
22:11 * Dark-Fx shrugs
22:11 < Dark-Fx> $SUPPORT_DEPARTMENT needs to get their shit together
22:12 < techieb0y> and no support contract, so you can't just RMA them?
22:12 < Dark-Fx> Probably out of warranty
22:12 * Dark-Fx shrugs.
22:12 < Dark-Fx> I haven't been around long enough to know.
22:12 < techieb0y> yeah, I guess they are 4+ years old now
[ right here is when I emailed the helpdesk monkey mailing list asking why rock was down ]
22:14 * NUXI ducks
22:17 < aragirn> NUXI, troll
22:17 < NUXI> :P
22:18 < NUXI> i'll concede that
22:33 < NUXI> no one replied :(
22:38 < NUXI> so, when are you guys gonna find your server?
22:41 < Dark-Fx> NEVER


So 10 hours later the machine is still MIA.

Comments

[dark-fx.livejournal.com]

Ugh, I can't believe I wasted 3 hours looking for the machine and we still don't know for sure which one it should be, or where it was last at. $UNIX_SYSADMIN is off at $CAR_RACING_EVENT so we can't get a hold of him till it's over. Luckily the machine is only used for people trying to get into our department from off campus...

[jon787.livejournal.com]

Last I checked, rock was the only machine outside the subnet you can SSH into.

[techieb0y.livejournal.com]

by which you mean, it's the only machine you can SSH into from outside the subnet. Last I checked, it was in fact inside the subnet. (Well, until today, anyway...) :p

[jon787.livejournal.com]

Shut up :P and apparently im wrong anyway. Its just the one users are told about.

[vulpisfoxfire.livejournal.com]

Heee. Amusing story, though it's not the first time I've heard of a server becoming physically lost. The geek in me is suggesting that with the name 'rock', you might want to see if there are any professors named 'Light' or 'Wiley' on campus, and have a few words with them... ;-)

[jon787.livejournal.com]

its cause the subdomain is .me, so the name is rock.me

Others we've had are:
show.me
lead.me
hack.me (openbsd honeypot)
rescue.me
Probably more im forgetting about

[kalium.livejournal.com]

just.kill.me

I read this and thought "Marilyn Manson was right - rock is dead".

[fearrett.livejournal.com]

Clearly you need steal.me


Our subdomains are all either LOTR nouns or simpsons characters.

that, or horrible movies.

We do in fact have a "Reefer-madness" server

[jecook]

lol.

the machines on sub-ether.net are all named after astronomical things:
universe.sub-ether.net
galaxy
pulsar
wormhole
whitehole (bonus points for guessing what I use it for)
comet, the incredible traveling laptop (of DOOMZORS!!!11oneoneone)
etc.

[wibbble]

If you ever find it, it should be renamed to 'find.me'.

[xforge.livejournal.com]

what, no "bite?"

[emsporter.livejournal.com]

My partner lost a laptop once. Physically lost; was responding to ping, but he didn't know where in the apartment it was. Took him a couple days to find it.

[jcaswell.livejournal.com]

We have that with a desktop machine at school. We know where it *should* be, and it responds to ping, but it isn't there. It's been "missing" for about 4 years now :)

[the-s-guy.livejournal.com]

Isn't the classic solution to that problem to remote into the machine, turn up its audio to max, and start running through its sound files?

I'm guessing that the old standby of "trace the cables" doesn't work too well with WiFi.

[spooforbrains.livejournal.com]

Or if it has no sound hardware, run a find / and listen for crunching noises ...

[the-s-guy.livejournal.com]

Wot, no onboard beeper, even?

[siliconshaman.livejournal.com]

BTDT [with a wifi router/switch]

Round about that point you seriously start considering trying a forked hazel twig...

Thankfully, most things still need power cables, so start at the outlet and work your way back...
Although, if the MIT boys ever get their bright idea of wireless power working, we'll be well and truly up the creek.

[spooforbrains.livejournal.com]

Did he submit the story to bash.org?

[jahbulon.livejournal.com]

Mmmm...

#5273 +(29282)- [X]

hm. I've lost a machine.. literally _lost_. it responds to ping, it works completely, I just can't figure out where in my apartment it is.


Interesting similarity in word choice there, eh?

[mogaribue.livejournal.com]

If you have access to a core router, find the arp address, then you can find the switch its connected to via the cam tables. Even if its off, the timeout is usually 4 hours on arp.

[spooforbrains.livejournal.com]

It's this kind of comment that makes me feel like a rank amateur in the tech world.

[mouser.livejournal.com]

I know what phaedrus is talking about - it's not an optimal plan in many network configurations anyway.

[dark-fx.livejournal.com]

Trust me, we tried this. Arp timeouts usually aren't more than 5-10 minutes on most boxes. Machine was long gone by the time we thought to check it.

[fenrirvallin.livejournal.com]

I just wanted to say that this post made my night.

Makes me miss working for the government. :(

[kuang.livejournal.com]

I had a 24 port HP switch vanish into nowhere, quite literally. When I'd worked out which one it was and visited, the ceiling tiles were up and there was a perfectly switch sized hole between the brackets. It's nice to be able to say that a piece of kit is off the network, and mean it for a change :)

[spooforbrains.livejournal.com]

So, what happened to it?

[kuang.livejournal.com]

Never did find out. I think somebody decided they wanted one, found the room unlocked (which it absolutely shouldn't have been, but you know teachers..) and helped themselves.

TBH I was glad to see the back of it as it frequently locked up and lots its VLANS, but still :)

[spooforbrains.livejournal.com]

Hah, well at least they got themselves a duff switch ...

[arabwel.livejournal.com]

Pretty stellar for you guys *grin*

*resists the urge to use this in a story*

[canray.livejournal.com]

"Can I haz Rock plz?"

Follow Up Email From $UNIX_SYSADMIN

[mtupyro.livejournal.com]

Aha, $FEARLESS_LEADER! I see you have found my secret lair.


But which machine is the REAL rock.me?


You will never figure it out! Bwahahahahahaha!


Rock.me is back up and at least plugged into the correct network jack if
still not actually labeled correctly.


-$UNIX_SYSADMIN

Re: Follow Up Email From $UNIX_SYSADMIN

[manuka.livejournal.com]

Ah, I see the mark of a true BOFH there :)

Re: Follow Up Email From $UNIX_SYSADMIN

[manuka.livejournal.com]

"It's right where it ought to be, it's just not labeled correctly. Figuring the rest out is left as an exercise to the reader"

[manuka.livejournal.com]

I remember a story (but I'm a little fuzzy on the details) a while back about a Novell server that had been up for years and eventually developed a hardware failure of some sort, and they had to trace it down by following wires, eventually finding that the closet where it was located had been completely sheetrocked in on all 4 walls. The remodel had been done several *years* before, and very few of the staff that were present for the remodel were still around.

[echan.livejournal.com]

[from the post]
22:04 < NUXI> techieb0y: you obviously have never heard of the netware server that got sealed into a wall
22:04 < techieb0y> that I've heard of, yes.
22:04 < NUXI> http://www.techweb.com/wire/story/TWB20010409S0012 (http://www.techweb.com/wire/story/TWB20010409S0012)
22:04 < NUXI> for those that haven't heard it

[manuka.livejournal.com]

ah, musta missed that on a page scroll... :)

[hopefulnebula]

...wow.

Bonus points for the Short Circuit reference, though.

[ptstech.livejournal.com]

I was having a very bad day - until I read this...