(no subject)

[ihateemo.livejournal.com]

This morning there's a witch hunt going on for someone who added a VERY BAD CONFIG LINE this morning.

It's quite funny to watch everyone using every trick in the book to catch the culprit (I was a suspect because I was VPN'd in around that time), without resorting to TACACS logs.

Oh yeah, we don't have TACACS. Weeeeee!

(And no, it wasn't me. :D)

Comments

[major-error.livejournal.com]

what line would that be?
rm -rf ...?
format c: /u/c/v:"" ...?

[ihateemo.livejournal.com]

Null routes. On our core switches. Whoops!

[jecook]

WhooHoo!!!

When I worked at [isp], we had a few implementation engineers do Bad Things. They didn't get written up, but they were mocked mercilessly.

things like:

"debug all" on a 7513 (fully loaded, too. It cherry bombed the router hard)

route ip 0.0.0.0 SerialX (customer's T-1 port on the router) This didn't cherry bomb the router, but it did choke the customer's T-1 until it got fixed. The Engineer had a banner placed above his desk that stated "We bring the Internet to YOU!" for a couple weeks after that.

I did my own bad thing there (i.e. I got into an accident whilst traveling and crashed the rental car) and was mocked as well. I don't *want* to know where the blow-up doll came from, though....

[whitewolf3399.livejournal.com]

Thankfully I've so far been spared that type of fun...

Unlike the analyst who turned did this on a juniper core router:

edit interfaces so-1/3/0.0
set disable
top
edit interfaces so-1/3/1.0
set disable
top
commit and-quit

That router was our 'edge' to UUNET/Level3... The interface's were the link back to our core hub in Miami FL. Not only did he kill internet access for a large portion of the Miami and surrounding area but he locked everyone out of the router :)

[jecook]

Ouch. Only way to fix that is a serial console connection...

[whitewolf3399.livejournal.com]

Yes, he got a surprise two week vacation for that one... luckily we have in-band access to the console port, it's just not published info

[ihateemo.livejournal.com]

"We bring the Internet to YOU!"

Hahahaha!

[kizayaen.livejournal.com]

But I neeeeeeed TACACS. I need it or I will explode.
That happens to me sometimes.

[wyronth.livejournal.com]

Sir or Madam, I love you a little bit now.

[kizayaen.livejournal.com]

Sir's fine, thanks. ;)

[zastrazzi.livejournal.com]

And that ladies and germs, is why we set log hosts on all devices :)

Cuz when we blow something up but good, we want to be able to point and laugh at the poor bastid who did it *grin*

Might want to introduce the network lads and lasses to archive config though. Handy for quick rollbacks AND for pointing fingers!