When big organizations go bad.

[emsporter.livejournal.com]

Backstory, which is probably necessary. Awhile back, now - this story happened nearly two years ago - I lost my job practically overnight. Aware of the need to eat and pay bills, I took the first job I could find that would cover said bills, despite it's crappiness (helpdesk in a callcenter). We were firstline support for the organization in question, ticket handlers, and generally treated like the muppets most of my coworkers were. I've also posted this story before over at TSC.

I can deal with run of the mill stupidity. However, this problem sticks in my mind as the worst I've ever dealt with.

I first encountered the issue as an incoming email arrived in the call handling software in my slot. Customer was angry. Very angry. By this stage, having been there about two months, I was thoroughly aware I worked for an organization full of muppets, so I delved into his past call notes. Customer can't send email to a business associate at CompanyB. Various folks have checked the email address validity, checked that Customer can send email both internally and externally - it's just that mail to CompanyB doesn't get through. (I worked for CompanyA. The parent company of CompanyA also owned CompanyB.)

Getting to this point, between Helldesk troubleshooting and the customer going on leave for a week, took us three weeks.

The call got punted to the mail system admins. Nope, our exchange servers are working fine, farkoff. Issue gets punted back to Helpdesk for reassignment. (Four weeks.)

Helpdesk punt the issue to the messaging team. (This team deals with communications between the company infrastructure and the rest of the world. I never really worked out what they did.) Said team say it's not their problem, return to helpdesk for reassignment. (Five weeks.)

Helpdesk punt issue to general server admins. Server admins work out the problem - a basic piece of information, freely available in the original bounceback messages from CompanyB. Company A’s mail server has no reverse DNS records. Company B thus (quite justly) rejects their email. Customer sees this as “My email is broken when I email Company B, please fix it NOW.” The server admins don't handle DNS support, a networking team do.

We're now at week seven of this issue being known, and this was the point where I received the customer's umpteenth follow-up mail. Given that I'd just left a job at a web-support place, I had a pretty functional understanding of DNS. I explained the issue as simply as I could to the customer, and receive a glowing email in return.

Hi Mahal.

Thanks for your update - it’s by far the most helpful response I’ve ever received from helpdesk. If there’s a reward for sending humanized, non-templated, useful and explanatory emails, you should get it.

Apparently, this was the first time the caller has ever had his issue explained. He was angry because he thought our admins were too lazy to fix a small issue with his personal email. If someone had explained it was a greater problem, he would have been less upset.

I neglected to explain to him that the problem could have been fixed immediately, had competent helpdesk staff, competent email admins, or competent network admins actually read the error message.

I was fuming, at this point. The issue was obvious to anyone with the ability to read error messages and spend 5 minutes on google. However, due to various bureaucrappic rules, I wasn't allowed to reassign the ticket to the correct team myself. Instead, I had to go to the helpdesk member who had said ticket in their queue, to suggest the correct assignment. I tried to pass this on to the agent handling. His response? “Well, if it doesn’t belong with the group it’s with now, they’ll send it back. Don’t worry!”

Of course, a new agent couldn’t possibly know anything about technical issues. Never mind that he doesn’t know dick about what I worked in before being dumped in helldesk. I did try to explain what DNS was, and how I knew that was the problem. However, when he asked what an IP address was, I gave up and reassigned the damn ticket myself.

I left the job about six weeks later, for greener pastures. The job still wasn't resolved. CompanyA still didn't have reverse DNS for their mail servers. Customer still can't mail his customer/contact/coworker at CompanyB.

All because no-one with the ability to do it knows a DNS record needs to be updated. One five minute fix, maybe 24 hours replication time, max, waiting on 13 weeks of cranio-rectal inversion.

Comments

[jimbojones.livejournal.com]

It's disturbing how few people know DNS... especially how few people know how to implement DNS.

Frequently, including the very people who are supposed to be doing it.

Most mailserver admins don't know jack shit about SMTP, either. Sigh.

[mouser.livejournal.com]

It's disturbing how few people know DNS... especially how few people know how to implement DNS... but MOST expecially the number that don't know anything about DNS that are in CHARGE of the servers that deal with it.

[jecook]

I think part of that reason is because once you've implemented it, it just works, and unless things change, you don't have to futz with it.

I think the last time I had to update sub-ether's zone file was.... 2006. and my reverse DNS is handled by my ISP, but it's for a single ip address, so that's no biggie there.

[jecook]

as for smtp... I know diddly and squat about it, so I'm one of those bastids...

DNS I have something of a clueon about, enough to implement it without breaking stuff, and to make sure I don't break other people's stuff.

[mouser.livejournal.com]

Yes, but you know how to read an error message and LOOK IT UP.

Seriously, that's like 85% of what we do. (5% is "Just re-boot the damned thing!")

[vortex.livejournal.com]

I worked in a NOC (Network Operations Center) for a couple of large companies and I can assure you that once DNS is set up, there is a lot of "maintenance" to be done, considering that Large companies have thousands of IP addresses, some facing the Public (Internet facing) and Internal (inside the firewalls).

But, like the OP said...5 minutes max to correct the A or B records to fix the described problem...

[ghostdandp.livejournal.com]

Feel very lucky.

Worked for a ISP for 5 years.. I still have nightmares about BIND, MX records, and A lines.

[jecook]

Oh, I hear you. The planning and implementation guys are who taught me what I do know about DNS and unix sysadmining when I worked for Global Crossing.