![[identity profile]](https://www.dreamwidth.org/img/silk/identity/openid.png){kind=small}
coyoteden.livejournal.com posting in ![[community profile]](https://www.dreamwidth.org/img/silk/identity/community.png){kind=small}
techrecoveryA cut-rate telecom company who shall remain nameless has put a client of mine through absolute hell over the past week.
It all started last week. They aren't technically inclined, all they knew was their T1-supplied internet got so slow it was unusable. They called said company, who walked them thru a few totally pointless troubleshooting steps and determined it was an internal problem. That's when I got called. After resolving a couple of unrelated LAN issues, I dived into why their T1 was sucking muchly.
Hmm... sites resolve fine, ICMP ping is fine with low latency, but make a TCP connection and 90% of the packets never make it back to our router. No T1 alarms. A few sites DO work, but they are few and far between. Traceroute shows no patterns.
Time to call the dreaded tech support. One ringyding... Two ringydings.. .and finally, after an hour and a half of the most godawful canned music I've ever been subjected to...
<short version>
Hello tech support? We've got a problem, and it's not our T1. I can resolve, ping and traceroute, but as soon as I make a connection to any site, any protocol, anything... It's unbearably slow or it times out.... DNS? Err, the sites resolve fine, I just.... OK,OK, I'll try different DNS servers... Nope, no difference... Yes I've tried it from different machines, actually, I'm testing it from the router... it's not our end, I'm positive... Wait, you want me to do what? And call back? Can't we just skip this, I mean.. It's NOT a LAN problem!
They refused to listen to me... so I hang up, strip things down to one machine with a static IP, a crossover cable, and the T1 box. While I'm at it, how about I just bend over and lube up for you thickheaded script-spewers. Oh, the humanity.
So I call back and wait another hour to talk to somecritter....
Hello? Look, I have ONE machine plugged directly into the T1 and it's probably getting wormed as we speak. Same problem. Finds the site, pings it fine, connects and times out... A FEW sites work. What... DNS Servers? Yes, the last guy had me change them to... Excuse me? Which ones should I be using? Those were the ones I had to start with. You guys just had me change them! Change them back???
brain-damaged lemurs.
Listen up.. you guys have a problem. It's not the DNS. Could I please talk to someone in your NOC? I'll give you a cookie....
...another hour on hold passes... NOC? ExplainExplain... We can connect to sites but it just times out... It's not our LAN, your tech support and I have ruled that out.. It's definitely not a name server problem, cause I can find the sites and ping them but if I...
EYAAAAAHHH! IF YOU ASK ME TO CHANGE THE DNS SERVERS ONE MORE TIME I'M GOING TO BITCHSLAP YOU THROUGH THE GODDAMN PHONE!!!
Thank you. Ahem... As I was saying, most sites time out. A couple work, and I can reach another customer of yours. It doesn't appear to be a routing problem, as the sites that do work are all over the 'net. My traceroutes show no pattern. Email you the traceroutes? Right away. Didn't I mention we can't get to our e-mail? Or anywhere, really? AT ALL? Yes, I can fax them to you.
So they give me a fax number. Of course it was the wrong number. I don't know what the hell a nursing home is going to do with a traceroute.
That was last week. I call the customer to follow up today, and they are still down. Oh gods. Back on site, back on hold for an hour with these bloody cretins. At least this time I can call the NOC directly. I explain the whole problem again, and after I indulge them with more DNS swappery, they suddenly decide to take a hint and forward the ticket to Engineering.
Beam us up Scotty. Now I know where they keep all their clue. Within 15 minutes I get call from a network engineer. He sticks a packet sniffer on the line and has me try a few sites that don't work followed by a couple that do. Seems the ATM router on their end is mysteriously refusing to pass packets to the bridge group we shared with a few hundred T1 customers. He cycles the interface and *poof!*
C.C.B.F.M.*
I've learned three things from this experience.
* Came Clear By Fucking Magic
It all started last week. They aren't technically inclined, all they knew was their T1-supplied internet got so slow it was unusable. They called said company, who walked them thru a few totally pointless troubleshooting steps and determined it was an internal problem. That's when I got called. After resolving a couple of unrelated LAN issues, I dived into why their T1 was sucking muchly.
Hmm... sites resolve fine, ICMP ping is fine with low latency, but make a TCP connection and 90% of the packets never make it back to our router. No T1 alarms. A few sites DO work, but they are few and far between. Traceroute shows no patterns.
Time to call the dreaded tech support. One ringyding... Two ringydings.. .and finally, after an hour and a half of the most godawful canned music I've ever been subjected to...
<short version>
Hello tech support? We've got a problem, and it's not our T1. I can resolve, ping and traceroute, but as soon as I make a connection to any site, any protocol, anything... It's unbearably slow or it times out.... DNS? Err, the sites resolve fine, I just.... OK,OK, I'll try different DNS servers... Nope, no difference... Yes I've tried it from different machines, actually, I'm testing it from the router... it's not our end, I'm positive... Wait, you want me to do what? And call back? Can't we just skip this, I mean.. It's NOT a LAN problem!
They refused to listen to me... so I hang up, strip things down to one machine with a static IP, a crossover cable, and the T1 box. While I'm at it, how about I just bend over and lube up for you thickheaded script-spewers. Oh, the humanity.
So I call back and wait another hour to talk to somecritter....
Hello? Look, I have ONE machine plugged directly into the T1 and it's probably getting wormed as we speak. Same problem. Finds the site, pings it fine, connects and times out... A FEW sites work. What... DNS Servers? Yes, the last guy had me change them to... Excuse me? Which ones should I be using? Those were the ones I had to start with. You guys just had me change them! Change them back???
brain-damaged lemurs.
Listen up.. you guys have a problem. It's not the DNS. Could I please talk to someone in your NOC? I'll give you a cookie....
...another hour on hold passes... NOC? ExplainExplain... We can connect to sites but it just times out... It's not our LAN, your tech support and I have ruled that out.. It's definitely not a name server problem, cause I can find the sites and ping them but if I...
EYAAAAAHHH! IF YOU ASK ME TO CHANGE THE DNS SERVERS ONE MORE TIME I'M GOING TO BITCHSLAP YOU THROUGH THE GODDAMN PHONE!!!
Thank you. Ahem... As I was saying, most sites time out. A couple work, and I can reach another customer of yours. It doesn't appear to be a routing problem, as the sites that do work are all over the 'net. My traceroutes show no pattern. Email you the traceroutes? Right away. Didn't I mention we can't get to our e-mail? Or anywhere, really? AT ALL? Yes, I can fax them to you.
So they give me a fax number. Of course it was the wrong number. I don't know what the hell a nursing home is going to do with a traceroute.
That was last week. I call the customer to follow up today, and they are still down. Oh gods. Back on site, back on hold for an hour with these bloody cretins. At least this time I can call the NOC directly. I explain the whole problem again, and after I indulge them with more DNS swappery, they suddenly decide to take a hint and forward the ticket to Engineering.
Beam us up Scotty. Now I know where they keep all their clue. Within 15 minutes I get call from a network engineer. He sticks a packet sniffer on the line and has me try a few sites that don't work followed by a couple that do. Seems the ATM router on their end is mysteriously refusing to pass packets to the bridge group we shared with a few hundred T1 customers. He cycles the interface and *poof!*
C.C.B.F.M.*
I've learned three things from this experience.
- This telco sucks enough to have an event horizon.
- Google really is impervious to evil. It was the only site that worked reliably, no matter what IP their round-robin AkaDNS handed me.
- I can kill my cable modem here at home just by sending it a few dozen 16K packets. Not encouraging at all.
* Came Clear By Fucking Magic
![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
no subject
Date: 2004-06-28 09:29 pm (UTC)no subject
Date: 2004-06-28 11:49 pm (UTC)clueless nocs
Date: 2004-06-29 12:27 am (UTC)On port 53.
Via TCP & UDP.
Machines being attacked: their DNS servers.
The IP addresses for the servers "attacking" them resolved to rns2.myisp.com. and rns3.myisp.com. I, being the lowly level 1 "I can't get my email but I'm sending an email to tech support" agent replied to the email asking which port THEY were using for their DNS, and exactly how much traffic they were getting per second, and the headers of one of the sniffed packets so I could forward the info to abuse. I never got a response. :(
I can perfectly understand users not knowing things like this; but a NOC?!.. sigh.
Re: clueless nocs
Date: 2004-06-29 06:10 pm (UTC)/sorry, couldn't resist.
Re: clueless nocs
Date: 2004-06-29 11:41 pm (UTC)I wasn't high or anything when I wrote that, I swear!
no subject
Date: 2004-06-29 05:01 pm (UTC)2. Google isn't a webpage. Google is magic. Nuff said.
3. AOL can kill my cable modem by installing dial-up adaptors under network properties. That's not encouraging either.
hahahaha
Date: 2004-06-29 07:17 pm (UTC)That's a classic.
Not my invention at all.
Date: 2004-06-29 10:59 pm (UTC)Re: Not my invention at all.
Date: 2004-07-01 08:17 am (UTC)