[identity profile] emsporter.livejournal.com posting in [community profile] techrecovery
Backstory, which is probably necessary. Awhile back, now - this story happened nearly two years ago - I lost my job practically overnight. Aware of the need to eat and pay bills, I took the first job I could find that would cover said bills, despite it's crappiness (helpdesk in a callcenter). We were firstline support for the organization in question, ticket handlers, and generally treated like the muppets most of my coworkers were. I've also posted this story before over at TSC.

I can deal with run of the mill stupidity. However, this problem sticks in my mind as the worst I've ever dealt with.

I first encountered the issue as an incoming email arrived in the call handling software in my slot. Customer was angry. Very angry. By this stage, having been there about two months, I was thoroughly aware I worked for an organization full of muppets, so I delved into his past call notes. Customer can't send email to a business associate at CompanyB. Various folks have checked the email address validity, checked that Customer can send email both internally and externally - it's just that mail to CompanyB doesn't get through. (I worked for CompanyA. The parent company of CompanyA also owned CompanyB.)

Getting to this point, between Helldesk troubleshooting and the customer going on leave for a week, took us three weeks.

The call got punted to the mail system admins. Nope, our exchange servers are working fine, farkoff. Issue gets punted back to Helpdesk for reassignment. (Four weeks.)

Helpdesk punt the issue to the messaging team. (This team deals with communications between the company infrastructure and the rest of the world. I never really worked out what they did.) Said team say it's not their problem, return to helpdesk for reassignment. (Five weeks.)

Helpdesk punt issue to general server admins. Server admins work out the problem - a basic piece of information, freely available in the original bounceback messages from CompanyB. Company A’s mail server has no reverse DNS records. Company B thus (quite justly) rejects their email. Customer sees this as “My email is broken when I email Company B, please fix it NOW.” The server admins don't handle DNS support, a networking team do.

We're now at week seven of this issue being known, and this was the point where I received the customer's umpteenth follow-up mail. Given that I'd just left a job at a web-support place, I had a pretty functional understanding of DNS. I explained the issue as simply as I could to the customer, and receive a glowing email in return.
Hi Mahal.

Thanks for your update - it’s by far the most helpful response I’ve ever received from helpdesk. If there’s a reward for sending humanized, non-templated, useful and explanatory emails, you should get it.
Apparently, this was the first time the caller has ever had his issue explained. He was angry because he thought our admins were too lazy to fix a small issue with his personal email. If someone had explained it was a greater problem, he would have been less upset.

I neglected to explain to him that the problem could have been fixed immediately, had competent helpdesk staff, competent email admins, or competent network admins actually read the error message.

I was fuming, at this point. The issue was obvious to anyone with the ability to read error messages and spend 5 minutes on google. However, due to various bureaucrappic rules, I wasn't allowed to reassign the ticket to the correct team myself. Instead, I had to go to the helpdesk member who had said ticket in their queue, to suggest the correct assignment. I tried to pass this on to the agent handling. His response? “Well, if it doesn’t belong with the group it’s with now, they’ll send it back. Don’t worry!”

Of course, a new agent couldn’t possibly know anything about technical issues. Never mind that he doesn’t know dick about what I worked in before being dumped in helldesk. I did try to explain what DNS was, and how I knew that was the problem. However, when he asked what an IP address was, I gave up and reassigned the damn ticket myself.

I left the job about six weeks later, for greener pastures. The job still wasn't resolved. CompanyA still didn't have reverse DNS for their mail servers. Customer still can't mail his customer/contact/coworker at CompanyB.

All because no-one with the ability to do it knows a DNS record needs to be updated. One five minute fix, maybe 24 hours replication time, max, waiting on 13 weeks of cranio-rectal inversion.

Date: 2008-03-07 05:37 am (UTC)
From: [identity profile] jimbojones.livejournal.com
It's disturbing how few people know DNS... especially how few people know how to implement DNS.

Frequently, including the very people who are supposed to be doing it.

Most mailserver admins don't know jack shit about SMTP, either. Sigh.

Date: 2008-03-07 05:49 am (UTC)
From: [identity profile] mouser.livejournal.com
It's disturbing how few people know DNS... especially how few people know how to implement DNS... but MOST expecially the number that don't know anything about DNS that are in CHARGE of the servers that deal with it.

Date: 2008-03-07 05:56 am (UTC)
jecook: (Default)
From: [personal profile] jecook
I think part of that reason is because once you've implemented it, it just works, and unless things change, you don't have to futz with it.

I think the last time I had to update sub-ether's zone file was.... 2006. and my reverse DNS is handled by my ISP, but it's for a single ip address, so that's no biggie there.

Date: 2008-03-07 05:58 am (UTC)
jecook: (Default)
From: [personal profile] jecook
as for smtp... I know diddly and squat about it, so I'm one of those bastids...

DNS I have something of a clueon about, enough to implement it without breaking stuff, and to make sure I don't break other people's stuff.

Date: 2008-03-07 06:01 am (UTC)
From: [identity profile] mouser.livejournal.com
Yes, but you know how to read an error message and LOOK IT UP.

Seriously, that's like 85% of what we do. (5% is "Just re-boot the damned thing!")

Date: 2008-03-07 06:40 am (UTC)
From: [identity profile] vortex.livejournal.com
I worked in a NOC (Network Operations Center) for a couple of large companies and I can assure you that once DNS is set up, there is a lot of "maintenance" to be done, considering that Large companies have thousands of IP addresses, some facing the Public (Internet facing) and Internal (inside the firewalls).

But, like the OP said...5 minutes max to correct the A or B records to fix the described problem...

Date: 2008-03-07 12:58 pm (UTC)
From: [identity profile] ghostdandp.livejournal.com
Feel very lucky.

Worked for a ISP for 5 years.. I still have nightmares about BIND, MX records, and A lines.

Date: 2008-03-07 01:31 pm (UTC)
jecook: (Default)
From: [personal profile] jecook
Oh, I hear you. The planning and implementation guys are who taught me what I do know about DNS and unix sysadmining when I worked for Global Crossing.

Date: 2008-03-07 06:02 am (UTC)
From: [identity profile] ohmyhead.livejournal.com
I'll admit I don't know TOO much about DNS, but when people in my company could not get through to AOL email addresses (personal friends), the complaints poured in. I saw some bounced messages, spent said 5 mins on Google, contacted the Postmaster at AOL, took her less than a minute to discover we had no reverse DNS, I grabbed the nearest network guru, stood over him while he fixed it. One day.

So sorry you had to go through that. I used to wonder why so many people were going postal/Columbine at their offices. After 10 years on a helpdesk, I no longer wonder.

Date: 2008-03-07 09:00 am (UTC)
From: [identity profile] kuang.livejournal.com
I remember the feeling of dread years ago when I ran a scavenge on a DNS server because nothing was resolving to the correct hostname first time. It worked a treat at first, and then the calls started coming in to ask why printers weren't working, etc. Seems that someone in the dim and distant past had decided that as from *NOW* dynamic updates would be turned off in some of the base images. No reason.

Date: 2008-03-07 10:08 am (UTC)
From: [identity profile] jerith.livejournal.com
One of the biggest bits of pain I've had to deal with recently is around DNS. The problem? The big third-party DNS provider we use has has a really crappy API. We can use their web interface, but that's not going to happen for thousands of entires at a time.

The main problem is that they take xml-rpc (which I don't like particularly, but it's about the best RPC standard available at the moment) and "extend" it in ways that break all existing client libraries. Their error reporting mechanism is crap, too: "X of the Y updates you requested succeeded."

Why can people not think before they do stuff like this?

Date: 2008-03-07 10:12 am (UTC)
From: [identity profile] brothersterno.livejournal.com
On a daily basis I am confronted by people who cannot read error messages. the amazing and wonderful thing about SMTP error messages is that they tell you *exactly* what the problem is, and if you can read the message you can solve the problem. And yet, I help people with these 1 or 2 sentence messages, and for some reason, they find them confusing.

Date: 2008-03-07 01:25 pm (UTC)
From: [identity profile] ghostdandp.livejournal.com
A few years ago I worked with a guy who actually developed a peice of software that would read error messages from SMTP servers and Apache and try to fix the problem itself. It intergrated with all the basics (BIND, the linux user system, Radius, etc). He made a pretty penny on it all too.

Date: 2008-03-07 12:03 pm (UTC)
ext_8716: (Servalan)
From: [identity profile] trixtah.livejournal.com
And this is why I refuse to work in an organisation where all steps of the call are routed via the Helpdesk. If I can't assign a job (as the mail gateway admin - no no no to internet-facing Exchange servers!) directly to the DNS/Firewall team, forget it.

However, the messaging team should have figured out themselves from the bounces, or doing a 30-second telnet to the server that wasn't accepting their mail (since you say the bounces were informative, that means the reject message from a telnet would have been informative as well). So your messaging admins are lazy. And incompetent, since you don't set up a mail server without having real hostnames for your mail gateway servers, OR proper PTR records that resolve to those hostnames (it frankly amazes me how many set-ups can't get those two simple steps right).

It's up to the mail admins to tell the server/DNS (however it's organised) admins what DNS records are needed - the mail, the SMTP RFCs and SMTP server best practice is their area of expertise, after all. If you don't know what's required for an internet-facing mail server (or any internet-facing service, actually), you shouldn't be administering one - it's not as if it's hard to learn about, either.

Date: 2008-03-07 01:21 pm (UTC)
From: [identity profile] ptstech.livejournal.com
I did some time at a couple of ISPs back in the day; I'm lucky in that our crew was more responsive than yours.

Oh yeah - ICON LOVE! *gank*

Date: 2008-03-07 05:01 pm (UTC)
From: [identity profile] jdotmi.livejournal.com
This just reminds me that the only reason I have really advanced as far as I have is my ability to google shit that I don't know. Because, seriously y'all, lookin' stuff up is hard!!11!1! o_O

Date: 2008-03-09 03:55 am (UTC)
From: [identity profile] blkrabbitofinle.livejournal.com
Grr. I suppose the Lotus Domino/Notes infrastructure for a small company. They started getting reverse DNS bouncebacks, so I emailed their tech guy to explain the problem and tell them they need to talk to their ISP to get them to set it up. The user was CC'd on this conversation, in the CC field- and decided to butt in and be incredibly rude to me, and ask me if the time he'd spend foofing around was chargeable back to my company because it was clearly Lotus Notes' problem.

I never knew Outlook can circumvent any security measures on servers that is has no control over to deliver your mail no matter what.

Date: 2008-03-09 04:31 pm (UTC)
From: [identity profile] jill-idle.livejournal.com
You might have better luck just emailing company B and politely asking they whitelist company A's mail server. I was like company B in this scenario once. It proved futile to get the my company A's reverse ptr set. As far as I know, to this very day, it's not set. In my company A, they have 4 mail servers. 3 were right, one was not, and no amount of begging seemed to matter.

Profile

techrecovery: (Default)
Elitist Computer Nerd Posse

April 2017

S M T W T F S
      1
2345678
91011121314 15
16171819202122
23242526272829
30      

Most Popular Tags

Style Credit

Expand Cut Tags

No cut tags
Page generated Jul. 20th, 2025 09:44 am
Powered by Dreamwidth Studios