Yay, bureaucracy

[ihateemo.livejournal.com]

Part of my job involves turning up ports on our switches for new server deployments. You'd think it would be easy enough - stick the port in the right VLAN, set speed and duplex, slap on a port description and turn up the port! 30 seconds worth of work, right?

Wrong.

In order to configure ports, we must first submit notification to do so, including a deployment plan (including our "proposed" configuration), a backout plan (in case something goes wrong...including config removal commands), which requires the approval of:

- The originator (ie. me)
- A secondary approver (usually the person requesting the configs)
- Our "change administrator" (some dude I've never even met)
- My direct manager
- His manager
- Our director
- Our VP

Change requests must be submitted AT LEAST five working days in advance of the change.

...oh but it gets better! If something goes WRONG (say, our cabling guys plug something into the wrong switch or server - which happens a LOT), then I must either open a ticket with the NOC - note that I have to call the NOC, not open the ticket myself - provide the ticket number to my manager, e-mail the ticket number to our VP requesting permission to change the port assignments/configs and then await him to e-mail the word "approved" to me before I can change three lines of config on the switch.

And this is only if the server is in production. Servers not in production must go through the 5-day approval process all over again.

Add to the fact that our SLA for creating deployment plans is 10 days (5 days to create the deployment plan taking into consideration our current workload, 5 days for the approval process), we could be talking almost a month just because someone wants me to fucking CONFIGURE A SWITCH PORT.

This is a Fortune 50 company as well. I wish I was making this shit up.

Comments

[manuka.livejournal.com]

Speaking from the standpoint of a network admin, in a large network like that, there's the serious possibility of major outages if someone down in L1 decides he's a network admin that day and buggers up the process because he thinks he knows how to do it after watching the guys in L2/L3 do it a couple of times.

In my network, it's no big deal because I'm the only cook stirring the pot. In a fortune 50 company, there are thousands of cooks in a very large pot. Having everything fully documented so that they can unfuck the network easily is a majorly important piece of the network management strategy.

The bureaucracy sucks, but the downtime sucks even more. Because you KNOW there's gonna be some idiot who thinks he knows what he's doing poking around in there. I've seen major corporate outages because someone was mucking about in the BGP tables who had no business doing so or bloody well ought to have known better.

[ihateemo.livejournal.com]

there's the serious possibility of major outages if someone down in L1 decides he's a network admin that day and buggers up the process because he thinks he knows how to do it after watching the guys in L2/L3 do it a couple of times.

L1s don't have the passwords (or the authority) to be making changes. Even the L2/L3 NOC guys have to page engineering for any changes/outages that require someone hopping on the routers. I'd say there are less than 20 people stirring the pot when it comes to the routing and switching side of things.

Documenting is fine - especially for major changes (one of my co-workers is implementing multicasting on two campuses) - but for a freaking port turn up? 20 days, if something goes wrong? Yowza.

[mogaribue.livejournal.com]

Yeah, that's just perverse. I don't care how many hands are stirring the pot, it's not that complicated. We have approval processes and change management meetings for things that may cause outages, but turning up a port is typically an email, if not a shout over the cube. And we've got about 12,000 people.

[vortex.livejournal.com]

I'm sure it has to more to do with certifications, such as Sarbanes Oxley, or the requirements for doing work with the Government, wrather than worrying about what L1 is doing.

I worked in the NOC for a Fortune 50 Internet Security company (recently purchased by IBM) and everyone on the NOC team as well as the Sys Admin team (ie. everyone who knows routers and switches) knows how to turn up ports blindfolded and in their sleep. It's the regulations that add alll of the steps that esentially end up making us extend the SLA's...

[gremlingirl.livejournal.com]

Until you said Fortune 50, I would have sworn you worked for my last job. Yeesh. Bad memories.

[ptomblin-lj.livejournal.com]

Let me guess - your company has ISO-9000 certification, right? This has that smell of "it doesn't matter how bad your processes are, as long as they're all documented in a binder and followed precisely" to it.

[canray.livejournal.com]

*Sniffs* YECH!!!

[vortex.livejournal.com]

Hooray fo Sarbanes Oxley! <---Sarcasm

I'm sure Sarbanes Oxley has a large part in your Fortune 50 Beurocracy...

[berkeleyfarm.livejournal.com]

When I worked for IBM Global Services, we had a daily change control teleconference, so our turnaround for routine changes (with ez backout) was very good.

But this was before SOX. (before y'all ask, I was in the public sector when it happened)

10-20 days for a port activation is seriously fscked up.

[vortex.livejournal.com]

I agree!

I worked for http://www.iss.net (note: recently purchased by IBM). Before I got fired for something I had no control over or access to, we completed Sarbanes Oxley...what a nightmare. I was glad to leave the company after that mess...

[snyperwolf.livejournal.com]

That sounds a lot like SBC ...

To buy a laptop for a workaround for a problem that they invented (I worked for a different company), they had to go get VP approval. It took like a month for them to go purchase a $400 dell laptop that they would probably never use, but we still had to configure. We only got it after swearing that we would never try to put it on the network.

[asbrand.livejournal.com]

Sounds like my last job at HellSouth.

Only...they're even worse. ;-)



-Az

[whitewolf3399.livejournal.com]

And it hasn't gotten any better...

[japester.livejournal.com]

I work at a nation wide ISP, and we go through the same procedure. Admittedly, with a few less people in the 'tick this box' category.
It's just me, my manager and the NOC manager.

As manuka said, if you fuck up on the wrong switch/router, the interweb goes away for some 10000+ people. That is wholy unacceptable, so you make sure that the customers never know what we're doing. That means you plan your work. You prepare. You make sure it happens the right way, every time. Sure it's a bitch when you're only enabling one switch port, possibly for a server that's not yet in production. I am waiting for one myself at the moment. It's part of the cost of doing business, and if your SLAs do not allow you the time to plan and accomodate the red tape, maybe they need to change.

[jjjiii]

Gee whiz. I sure hope you get paid by the hour!