Where did my server go...

[jon787.livejournal.com]

So some guys I used to work with lost a server today. It disappeared from the network at around 2pm and they can't physically locate it to fix it.


Proper credit where credit is due:
adam|beer is adamjaskie
Dark-Fx is dark_fx
pyro|lappy is mtupyro
techieb0y is techieb0y
NUXI is jon787 (me)
Not sure if anyone else who shows up is on LJ

13:50 -!- adam|beer [$USER@rock.$DEPARTMENT.$UNIVERSITY.edu] has quit [Ping timeout]
13:50 -!- Dark-Fx|rock [$USER@rock.$DEPARTMENT.$UNIVERSITY.edu] has quit [Ping timeout]
14:02 < Dark-Fx> Who killed rock
15:12 < pyro|lappy> lol
15:12 < pyro|lappy> $FEARLESS_LEADER physically cant find rock
15:12 < MotS> isn't it in the fishbowl?
15:13 < Dark-Fx> WE HAVE NO IDEA WHERE ROCK IS!
15:13 < Dark-Fx> heh
15:13 < pyro|lappy> lmao
15:13 < pyro|lappy> and no cell number for $UNIX_SYSADMIN
15:16 < Dark-Fx> There *is* a machine missing from where rock is *supposed* to be
15:17 < MotS> so there *isn't* a machine where rock is supposed to be?
15:18 < Dark-Fx> MotS: I have a feeling rock is now one of the unlabeled tower machines in the chem basement
15:18 < techieb0y> ...rock is missing?
15:18 < NUXI> pyro|lappy: NNN-NNNN
15:32 < pyro|lappy> NUXI: that's his home number
15:35 < NUXI> he probably doesn't have a cell
15:41 < MotS> dang
15:41 < Dark-Fx> Rock might be in the NOC
15:42 < MotS> Noc your rock?
15:42 < MotS> kinky?
15:57 < Dark-Fx> I think I found rock
15:57 < Dark-Fx> Rock and mandan have bad power suplies
15:58 < Dark-Fx> Rock is the old Askel
15:58 < Dark-Fx> Which isn't askel anymore, I can't ping it at least.
16:00 < Dark-Fx> Oh god that's hoooooooorible
16:01 < pyro|lappy> what room are you physically in?
16:01 < Dark-Fx> $DAS_BUNKER?
16:01 < pyro|lappy> MEEM?
16:01 < HuskyBot> i guess MEEM is really boring...
16:01 < Dark-Fx> No, chem
16:02 < Dark-Fx> Rocks PSU fan is making the worst noises ever.
16:02 < pyro|lappy> it can make a sound that sounds like brittney spears singing?
16:10 < pyro|lappy> Dark-Fx: why isnt rock up yet
16:11 < Dark-Fx> It's brokzored
16:11 < pyro|lappy> $USER_SUPPORT_MANAGER: "that's not a good sign"
16:11 < Dark-Fx> Okay, this isn't rock -$FEARLESS_LEADER
16:12 * Dark-Fx heards to noc
16:12 < pyro|lappy> "uh oh"
16:18 < Dark-Fx> lol where the hell is rock
16:19 < Dark-Fx> I spies a ruckus!
16:19 < techieb0y> ...how does one /lose/ a 4U box?
16:19 < Dark-Fx> We're not sure if it's a 1U or 4U
16:19 < techieb0y> didn't rock get transitioned to one of the V440s?
16:20 < Dark-Fx> I thought so
16:20 < techieb0y> and, do you know rock's MAC?
16:23 < Dark-Fx> not the current one :(
16:24 -!- agmlego [agmlego@$HOST.resnet.$UNIVERSITY.edu]] has joined #UNIVERSITY
16:25 < agmlego|frondeur> So, we're missing a server?
16:27 < Dark-Fx> completely have no idea where it's at.
16:28 * Dark-Fx talks with $COMPUTER_SECURITY_GUY
16:36 < Dark-Fx> back in chem
16:36 < Dark-Fx> Okay, now we think the machine is what arnheim used to be...
16:44 < Dark-Fx> When's the last time anyone's run uname -a on rock?
16:44 < Dark-Fx> Can you tell me what the machine was?
16:46 < agmlego|frondeur> So, what's the deal with rock? It's down, and you're trying to find it to fix it, or what?
16:47 < techieb0y> yeah, it's not responding to ssh, and somehow they seem to not know where it is.
16:47 * techieb0y isn't sure how that's possible
16:48 < agmlego|frondeur> techieb0y: Well, there are a lot of computers on campus...
16:48 < pyro|lappy> Dark-Fx: $SUPPORT_MONKEY_1 says that DNS resolves rock.me, so just start at the network cable and follow the packets
16:48 < pyro|lappy> =P
16:48 < agmlego|frondeur> ...one could probably just hide in a corner somewhere and pout without anyone noticing it.
16:48 < techieb0y> yeah, but servers don't tend to move often.
16:48 < techieb0y> so you'd think you're remember where you moved it to.
16:49 < Dark-Fx> pyro|lappy: DNS stuff is all unlocked for the servers though
16:49 < agmlego|frondeur> Maybe it came alive?
16:49 < Dark-Fx> techieb0y: this is $UNIX_SYSADMIN's failure
16:49 < pyro|lappy> maybe rock got bored and got up and left
16:49 < techieb0y> heh
16:49 < agmlego|frondeur> That's what I'm saying. Pulled a Johnny 5...
16:49 < techieb0y> the concensus is that it's not sn $DAS_BUNKER?
16:50 < agmlego|frondeur> "Johnny 5 is alive!"
16:50 < Dark-Fx> techieb0y: it might be
16:50 < techieb0y> $DAS_BUNKER isn't that big, how can you lose a machine in there?
16:50 < techieb0y> (and if it isn't in there, where else would it be?)
16:50 < Dark-Fx> techieb0y: NOC, $FEARLESS_LEADER mentioned something about REL
16:52 < techieb0y> why'd stuff get moved there?
16:52 -!- pyro|lappy [pyro@$HOST.rovernet.$UNIVERSITY.edu] has quit [Quit: Ex-Chat]
17:11 < Dark-Fx> still no rock!
17:11 < Dark-Fx> cataloged most of the machines in here
17:19 < Dark-Fx> Awesome. still. no. rock.
17:20 < agmlego|frondeur> Rock, why have you forsaken us?
17:20 < Byan> what happened to rock?
17:20 < agmlego|frondeur> It's missing.
17:22 < Dark-Fx> I think it's in the NOC.
17:22 < Dark-Fx> I don't have swipe :(
17:22 < agmlego|frondeur> What about the guy you have with you, or isn't there one?
17:22 < Dark-Fx> $FEARLESS_LEADER does
17:23 < Dark-Fx> We booted the 2 machines I suspected it was, but we might not have waited long enough for it to become pingable
17:26 < Dark-Fx> Anyone up here HAM?
17:26 < Dark-Fx> call up $UNIX_SYSADMIN_CALLSIGN and ask him where rock.$DEPARTMENT.$UNIVERSITY.edu physically resides.
17:42 < Dark-Fx> aragirn: you around?
17:43 < john> I stolz rock
17:43 < john> har har har
21:46 < aragirn> Dark-Fx, what's up?
21:56 < Dark-Fx> aragirn: I was going to ask you to go turn on the machine in the NOC labeled "allouez"
21:57 < aragirn> oh, sorry, I've been off work since 4pm
21:57 < Dark-Fx> yeah, not a problem
21:57 < Dark-Fx> Well, not MY problem.
21:58 < techieb0y> rock still missing?
21:58 < Dark-Fx> Yep!
21:58 < aragirn> how the hell do you lose rock?
21:58 < Dark-Fx> hahahaha
21:58 < techieb0y> wow.
21:58 < Dark-Fx> Because the machine isn't labeled as such
21:58 < Dark-Fx> We have 3 allouez.
21:58 < Dark-Fx> 2 are in the NOC, one is -spare
21:58 < techieb0y> but you think you know where it is, it's just off? Or are you still unsure where it is, physically?
21:59 < Dark-Fx> techieb0y: I'm pretty sure it's one of the allouez in the NOC.
21:59 < techieb0y> ...why are there more than one?
21:59 < Dark-Fx> I think we didn't wait long enough for the machine to fsck before it started to respond to pings.
21:59 < Dark-Fx> One is allouez, the other is allouez-spare.
21:59 < Dark-Fx> Neither of which are the real allouez.
21:59 < techieb0y> lol
22:00 * techieb0y recomends you buy a labelmaker and a putty knife.
22:00 < aragirn> Dark-Fx, and they were labeled rock and rock-spare when they were brought in
22:00 < Dark-Fx> really?
22:00 < Dark-Fx> On the tops of them or something?
22:00 < aragirn> no, that's what $UNIX_SYSADMIN told us they were
22:00 * Dark-Fx has no idea what $UNIX_SYSADMIN was thinking
22:00 < NUXI> Dark-Fx: job security
22:00 < aragirn> then we got a call, "Those are actually allouez and allouez-spare"
22:01 < Dark-Fx> No... Allouez is still in $DAS_BUNKER in checm
22:01 < Dark-Fx> chem
22:01 < MotS> wacky.
22:01 < Dark-Fx> methinks $UNIX_SYSADMIN was confused.
22:01 < Dark-Fx> Or we /really/ don't have a Rock.
22:01 < aragirn> lol
22:02 < NUXI> rock was stolen by underwear gnomes
22:02 * techieb0y still boggles at the idea of loosing a machine in a data center that small
22:03 < aragirn> techieb0y, it was never brought into our data center
22:03 < techieb0y> exactly.
22:03 < techieb0y> $DAS_BUNKER isn't big enough to loose anything in.
22:04 < NUXI> techieb0y: you obviously have never heard of the netware server that got sealed into a wall
22:04 < techieb0y> that I've heard of, yes.
22:04 < NUXI> http://www.techweb.com/wire/story/TWB20010409S0012
22:04 < NUXI> for those that haven't heard it
22:06 < Dark-Fx> techieb0y: /none/ of the machines in $DAS_BUNKER are rock.
22:06 < Dark-Fx> That much we know.
22:06 < techieb0y> ...then where is it?
22:07 < Dark-Fx> Missing...
22:07 < Dark-Fx> There's a spot where cables for rock used to go, at least according to the telcom info.
22:07 < techieb0y> when did it disapear?
22:07 * NUXI looks at his pretty sunfire v100
22:07 < Dark-Fx> It wasn't there.
22:07 < NUXI> its shiny
22:07 < Dark-Fx> I timed out at 13:50
22:08 < Dark-Fx> NUXI: when'd you get a v100?
22:08 < NUXI> Dark-Fx: :P
22:08 < Pyro> maybe that's why rock is down
22:08 < Pyro> someone stole it
22:08 < NUXI> rock was a v100 last I checked 4 years ago
22:08 < NUXI> not idea what model it is now
22:08 < techieb0y> rock was a v120 I thought?
22:08 < techieb0y> then it was migrated to a V440
22:08 < Dark-Fx> Should still be a 1U
22:08 < NUXI> i thought it was a v100 like herman was
22:09 < Dark-Fx> techieb0y: I think it was momentarily a V440 because $UNIX_SYSADMIN brought it up as the wrong box.
22:09 < techieb0y> oh
22:09 < Dark-Fx> I at least have IRC logs of it last being that Feb of this year.
22:09 < Dark-Fx> Only time I could find though
22:09 * MotS cackles.
22:10 < Dark-Fx> The only V440s we had were machines that reported to be what was labeled.
22:10 < techieb0y> and you've accountd for all 4 of those?
22:10 < Dark-Fx> yeah
22:10 < techieb0y> hmm
22:10 < Dark-Fx> Even though askel isn't used at all anymore.
22:10 < techieb0y> heh
22:11 < techieb0y> that's a pretty powerful doorstop, then.
22:11 < Dark-Fx> $FEARLESS_LEADER said it was end of life.
22:11 < Dark-Fx> One of the PSUs in it is dead
22:11 < techieb0y> oh
22:11 < Dark-Fx> One of the PSUs in Mandan are dead too though
22:11 * Dark-Fx shrugs
22:11 < Dark-Fx> $SUPPORT_DEPARTMENT needs to get their shit together
22:12 < techieb0y> and no support contract, so you can't just RMA them?
22:12 < Dark-Fx> Probably out of warranty
22:12 * Dark-Fx shrugs.
22:12 < Dark-Fx> I haven't been around long enough to know.
22:12 < techieb0y> yeah, I guess they are 4+ years old now
[ right here is when I emailed the helpdesk monkey mailing list asking why rock was down ]
22:14 * NUXI ducks
22:17 < aragirn> NUXI, troll
22:17 < NUXI> :P
22:18 < NUXI> i'll concede that
22:33 < NUXI> no one replied :(
22:38 < NUXI> so, when are you guys gonna find your server?
22:41 < Dark-Fx> NEVER


So 10 hours later the machine is still MIA.