(no subject)

[brotherflounder]

Just got this email from the University of South Carolina's engineering department:

"The network problems that we had Tuesday night have replayed themselves 90-fold this afternoon starting at about 1:30pm. There was again extreme slowness on the College network causing an outage. It was traced to a network bridging loop in a different research lab this time. We disabled the connection to the lab and rebooted the router to restore the network to working order. The network was up and down while we were tracing the problem down. It has been up for about 1.5 hours now, and appears stable.

Again, please be extremely careful when using mini-switches in your offices and labs. One of these devices improperly connected can bring down our entire network. If you are not sure how to set up a mini-switch, contact your departmental IT administrator for help."

Oy vey. I shudder to think what kind of loops these professors put into their research.

Comments

[the-s-guy.livejournal.com]

You'd think the network would have mechanisms in place to detect and lock out this kind of thing.

I'm assuming a script which momentarily chops the noisier segment of the network into smaller and smaller binary tree halves until it can't narrow it down any further wouldn't be of assistance?

(Bonus points if it narrows down the exact port(s) causing the problem and disables them until manually reinstated.)

[orangecone.livejournal.com]

Depends upon the network gear. I would've thought (IIRC) -- at the switch level -- spanning tree would've stopped it. Otherwise, an intelligent routing protocol will disable the port.

[docjeff.livejournal.com]

One wonders why they allow the network to be so easily corrupted. That's not particularly bright imnsho.

[vxo.livejournal.com]

I've tried to start a network storm like this on my home LAN out of boredom...

How does this happen, exactly? I've tried bridging two ports on a switch, but nothing interesting happened. Do you have to bridge two switches together through another switch to summon the etherspooge?

Yes, I really am that easily amused.

[mouser.livejournal.com]

Multiple switches, hubs, and routers work best, bought at random by lusers with the cheapest and WEIRDEST ones first in line - also have the WAN and LAN sides all mixed.

Had a luser that brought in a LinkSys ROUTER and plugged it in backwards. On a Monday morning. For about two hours

Took me a while to figure out why the internal DNS was all fraked up.

[vxo.livejournal.com]

Haha... I've seen that one happen before. I think they should come with a warning label.

They're frighteningly quick to respond with a dhcpoffer, beating out most real software-based dhcp systems with thrilling speed... then doing really fucked up things once they exceed their pool of leases (configurable on things like dd-wrt, but fixed to 25 or 50 on most).

[hisamishness.livejournal.com]

The fun ones I've seen are those that offer 192.168.x.x addresses back to users on the corporate LAN. Most frustrating if you don't notice the subnet right off in the diagnostic steps. Then comes tracking it down...

[hisamishness.livejournal.com]

We had this happen at work. It brought down an entire campus. Imagine, if you will, a little 10/100 'smart' hub with delusions of grandeur. It sent out, as best I can recall, BGR packets identifying itself as top of the heap. Now, imagine all the Cisco gear hearing this, adapting their route maps accordingly, and shoving multiple gig link's worth of traffic through this little device.

We now have filtering enabled that shuts down most ports as soon as it sees more than one concurrent MAC address in packets travelling over a given port. I forget the term the network guys used for this filter...

[japester.livejournal.com]

yeah, it's called portfast. The switch 'interogates' the equipment at the other end of the cable and if it detects a routing loop, it disables the port.

Your network must have been fuxx0red to be listening to BGP routes from routers it doesn't know about. You can filter those packets out on all the ports _except_ the ones you know should have BGP traffic on them. That's just asking to b p0wned

[vorro.livejournal.com]

a little while after that fiasco, we also had a VP kill a ton of ports when he tried plugging his router into any port he could find. the router would show network activity for about a tenth of a second and stop. convinced the jack wasn't working (and he was correct, but not for the reasons he believed), he would go find another one that wasnt being used...

yeah, the network's a ton more idiot-proof than it used to be, but it's not that easy sometimes to idiot-proof the original idiot-proofing, I guess...

[blossomingfire.livejournal.com]

I'm so glad my users can barely turn the computer on....and that I didn't take the job in Engineering a few years back.

[startrekempress.livejournal.com]

I adore your icon.

Also. I feel a carefully measured proportion of your pain.

[ihateemo.livejournal.com]

Como pasa? That's the stupidest network design ever. Your engineers need to go back to networking school; if one rogue switch can bring the whole network down, you need to fire them and hire new ones. :)

The data centre I used to work for configured individual VLANs for all their users to prevent just this kind of thing from happening; they forgot to implement this on one switch, however, and said switch was brought to its knees by a broadcast storm when some dumbass disabled STP on a switch and plugged it in.

[shifuimam.livejournal.com]

When I still had my tech support job, this dumbass kid in the campus apartments had bought, second-hand, an iBook and Airport base station. He had no idea what the base station was for, just that the friend he bought the stuff from had told him to "plug it in, and you can get on the internet without wires".

He brought down his subnet. Three. Times. In one month. After I repeately told him that personal wireless access points or routers were prohibited. After I repeatedly told him that he was hooking up his base station incorrectly and it was trying to assign IP addresses to the other people in his subnet.

I finally told him that if he did it again, he was guaranteed to have his network connection terminated for the rest of the academic year. I think that finally made him quit.

However, this is why the residence halls/apartments are on an entirely separate network from the rest of campus - and split up from there. It used to be that some nob freshman in the dorm would kill half the campus network...