You have GOT to be kidding me...
Nov. 4th, 2008 04:30 pm![[identity profile]](https://www.dreamwidth.org/img/silk/identity/openid.png){kind=small}
greenmansgrove.livejournal.com posting in ![[community profile]](https://www.dreamwidth.org/img/silk/identity/community.png){kind=small}
techrecoverySo, I can NOT with any good conscience recommend an ISP with a two letter name in which one of the letters is X.
So, my client's split T1 line has been down. Since Friday at 10:30am, when I placed the initial service call. Not a DSL line. A T1. After the initial call, I called back because I hadn't heard anything, and was told that after some remote testing, a technician had been dispatched on site, had tested the equipment, and declared that the problem was in the line between the building and the local phone company's CO. I questioned these findings, as when T1's go down, they don't usually go down by having the data half of the split T1 dying, while still leaving some of the phone lines working. But they insisted that this was a problem with the phone company's line, and that all the hardware in the building had been tested and cleared. So, a trouble ticket had been opened with AT&T.
The initial call was placed at 10:30. I was told that a technician would have 4 hours to respond. I got the information blaming AT&T at 1:30, after I hadn't heard anything and called back on my own.
At 4pm, still having had no update from the ISP, I called back again. I was told that an AT&T technician had been assigned, and that they would be working on the problem. I know that AT&T doesn't usually work during evening or weekend hours, and it was now 4pm on Friday, so I asked if AT&T would be stopping work at 5, or continuing to work on the problem until it was resolved. The ISP support person put me on hold, came back, and assured me that the AT&T tech would be working until the problem was resolved, and that they (the ISP) would inform me as soon as they knew anything more.
At 8pm on Friday night, I called back for an update. I spoke to another technician who told me that AT&T never works nights or weekends, and that no work would be done on the problem until 9am Monday morning. I asked for the name of the technician that I had spoken to earlier in the day, and was told "I can't give you that information." I then asked to speak to a supervisor, and was told "My supervisor went home at 5." When asked if he was the boss on site, or the final person I could speak to, he said "Well, there IS a floor supervisor." When I asked to speak to the floor supervisor, my phone support guru here asked me "What could he do for you?" I told him that he could either escalate the problem, or take my complaint. "He's not in the complaints department," I was told. I requested again to speak to the floor supervisor, and was told by the phone guru that HE had the discretionary power to decide who got to speak to a supervisor.
I'm so glad that I got the name of the moron that I was speaking to, BEFORE we had that little exchange. He was told that Monday morning, I'd be taking up the matter again... From the corporate office of his company, rather than through the tech side.
Early the next morning we arranged for a temporary internet connection using a wireless broadband card, and a gateway mail server. (Note that I have strongly suggested to my client on a number of occasions before this, that we get them a second internet connection and a dual-WAN router, just to make sure things like this don't happen.) So, the client has basic internet connectivity and is able to send and receive email, but the ISP doesn't know that, and I'm not about to tell them, for fear that they'll decide that the issue isn't as dire as it is.
Monday morning I'm back at the client site bright and early. Getting all the workstations set up to use the temporary connection, and getting them access to their mail. Meanwhile, I'm also on the line with the ISP and they're telling me that they're trying to get AT&T on site to get the problem resolved. I call back repeatedly throughout the day, and and told each time that a tech from AT&T has been assigned, and will be there somewhere between 9am and 6pm to address the problem, and that my client has the highest priority for both XO and AT&T.
This morning, I'm back at the client site because they still don't have connection. I reboot the router again (as I did on Friday, just in case), with no change. Before I leave the building to start calling the ISP, I talk to building security and the building engineer, who both tell me that an AT&T tech had been in the building working on the lines at the demark at 11:30 the previous day. Even though long after that, I was still being told that my client's building was the technician's very next stop.
I call the ISP. They tell me that the tech hadn't made it to the site, but that he'd been scheduled for this morning. When confronted with the fact that he'd been on site the previous day, they get real quiet. I suggest that there is a communications breakdown between AT&T and the ISP, and that they'd better get it fixed right away, because as of this morning, the ISP's client had a T1 that had been down for 96 hours. They told me that they'd call AT&T and get it all straightened out. A little while later, I get a call telling me that the technician would be on site by 1-2pm this afternoon. Despite the fact that this whole thing was supposed to have already been escalated to the very top levels. I also point out to the ISP that my client is also their client, and that my client shouldn't have to pay me to kick their ass to keep on track with AT&T. They have no response better than "I agree."
At 2pm, I call the ISP supervisor that I've been working with to find out what's going on. He tells me he's going to call AT&T rght now for an update. 3 minutes later he calls me back to ask if the technician is still on site. I'm not onsite, but I call the office of the building, and am told that he's still there working. I call the ISP's supervisor back and confirm that he is. His response? "Ok...I just wanted to make sure that AT&T wasn't lying to me."
On a hunch, I call the office of the building back. As I explain who I am, I'm told that the technician has just left, and that he declared the line to be just fine.
I call the supervisor back, give him that bit of information, and tell him to call AT&T to confirm. 20 minutes later, I call the supervisor back, and tell him that he should get a technician out on site to doublecheck the first technician's work. At which point, he apparently looks at the trouble ticket history, and admits to me... "I don't see any indication that a technician was ever sent out on site."
Me: "Let me get this straight. Your support staff performed some perfunctory tests and decided that the problem was at AT&T's end, despite my arguments against. Then, no technician was ever sent out to test the physical hardware on site, but passed the problem on to AT&T, and then didn't follow up with AT&T until I started holding your hand and making you follow up."
SR: "uhm... yes."
Me: "I think that you should be getting a technician out on site to get this problem fixed right NOW. Or maybe YESTERDAY."
SR: "uhm...yes."
Me: " Why are you still talking to me?"
So, we've got a technician from the ISP coming out in about 1/2 an hour to test all the on-site hardware. I'm not holding my breath. Meanwhile, I'm collecting bids from other ISP for alternate connectivity sources.
So, my client's split T1 line has been down. Since Friday at 10:30am, when I placed the initial service call. Not a DSL line. A T1. After the initial call, I called back because I hadn't heard anything, and was told that after some remote testing, a technician had been dispatched on site, had tested the equipment, and declared that the problem was in the line between the building and the local phone company's CO. I questioned these findings, as when T1's go down, they don't usually go down by having the data half of the split T1 dying, while still leaving some of the phone lines working. But they insisted that this was a problem with the phone company's line, and that all the hardware in the building had been tested and cleared. So, a trouble ticket had been opened with AT&T.
The initial call was placed at 10:30. I was told that a technician would have 4 hours to respond. I got the information blaming AT&T at 1:30, after I hadn't heard anything and called back on my own.
At 4pm, still having had no update from the ISP, I called back again. I was told that an AT&T technician had been assigned, and that they would be working on the problem. I know that AT&T doesn't usually work during evening or weekend hours, and it was now 4pm on Friday, so I asked if AT&T would be stopping work at 5, or continuing to work on the problem until it was resolved. The ISP support person put me on hold, came back, and assured me that the AT&T tech would be working until the problem was resolved, and that they (the ISP) would inform me as soon as they knew anything more.
At 8pm on Friday night, I called back for an update. I spoke to another technician who told me that AT&T never works nights or weekends, and that no work would be done on the problem until 9am Monday morning. I asked for the name of the technician that I had spoken to earlier in the day, and was told "I can't give you that information." I then asked to speak to a supervisor, and was told "My supervisor went home at 5." When asked if he was the boss on site, or the final person I could speak to, he said "Well, there IS a floor supervisor." When I asked to speak to the floor supervisor, my phone support guru here asked me "What could he do for you?" I told him that he could either escalate the problem, or take my complaint. "He's not in the complaints department," I was told. I requested again to speak to the floor supervisor, and was told by the phone guru that HE had the discretionary power to decide who got to speak to a supervisor.
I'm so glad that I got the name of the moron that I was speaking to, BEFORE we had that little exchange. He was told that Monday morning, I'd be taking up the matter again... From the corporate office of his company, rather than through the tech side.
Early the next morning we arranged for a temporary internet connection using a wireless broadband card, and a gateway mail server. (Note that I have strongly suggested to my client on a number of occasions before this, that we get them a second internet connection and a dual-WAN router, just to make sure things like this don't happen.) So, the client has basic internet connectivity and is able to send and receive email, but the ISP doesn't know that, and I'm not about to tell them, for fear that they'll decide that the issue isn't as dire as it is.
Monday morning I'm back at the client site bright and early. Getting all the workstations set up to use the temporary connection, and getting them access to their mail. Meanwhile, I'm also on the line with the ISP and they're telling me that they're trying to get AT&T on site to get the problem resolved. I call back repeatedly throughout the day, and and told each time that a tech from AT&T has been assigned, and will be there somewhere between 9am and 6pm to address the problem, and that my client has the highest priority for both XO and AT&T.
This morning, I'm back at the client site because they still don't have connection. I reboot the router again (as I did on Friday, just in case), with no change. Before I leave the building to start calling the ISP, I talk to building security and the building engineer, who both tell me that an AT&T tech had been in the building working on the lines at the demark at 11:30 the previous day. Even though long after that, I was still being told that my client's building was the technician's very next stop.
I call the ISP. They tell me that the tech hadn't made it to the site, but that he'd been scheduled for this morning. When confronted with the fact that he'd been on site the previous day, they get real quiet. I suggest that there is a communications breakdown between AT&T and the ISP, and that they'd better get it fixed right away, because as of this morning, the ISP's client had a T1 that had been down for 96 hours. They told me that they'd call AT&T and get it all straightened out. A little while later, I get a call telling me that the technician would be on site by 1-2pm this afternoon. Despite the fact that this whole thing was supposed to have already been escalated to the very top levels. I also point out to the ISP that my client is also their client, and that my client shouldn't have to pay me to kick their ass to keep on track with AT&T. They have no response better than "I agree."
At 2pm, I call the ISP supervisor that I've been working with to find out what's going on. He tells me he's going to call AT&T rght now for an update. 3 minutes later he calls me back to ask if the technician is still on site. I'm not onsite, but I call the office of the building, and am told that he's still there working. I call the ISP's supervisor back and confirm that he is. His response? "Ok...I just wanted to make sure that AT&T wasn't lying to me."
On a hunch, I call the office of the building back. As I explain who I am, I'm told that the technician has just left, and that he declared the line to be just fine.
I call the supervisor back, give him that bit of information, and tell him to call AT&T to confirm. 20 minutes later, I call the supervisor back, and tell him that he should get a technician out on site to doublecheck the first technician's work. At which point, he apparently looks at the trouble ticket history, and admits to me... "I don't see any indication that a technician was ever sent out on site."
Me: "Let me get this straight. Your support staff performed some perfunctory tests and decided that the problem was at AT&T's end, despite my arguments against. Then, no technician was ever sent out to test the physical hardware on site, but passed the problem on to AT&T, and then didn't follow up with AT&T until I started holding your hand and making you follow up."
SR: "uhm... yes."
Me: "I think that you should be getting a technician out on site to get this problem fixed right NOW. Or maybe YESTERDAY."
SR: "uhm...yes."
Me: " Why are you still talking to me?"
So, we've got a technician from the ISP coming out in about 1/2 an hour to test all the on-site hardware. I'm not holding my breath. Meanwhile, I'm collecting bids from other ISP for alternate connectivity sources.
![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
no subject
Date: 2008-11-04 10:39 pm (UTC)no subject
Date: 2008-11-04 10:51 pm (UTC)that said X* seems more like VAX/VMS circa '83 than any other.
no subject
Date: 2008-11-05 01:46 am (UTC)no subject
Date: 2008-11-05 10:34 am (UTC)We now have a 1Gb connection to a completely different company, and it costs us about half the amount of the aforementioned 100Mb V... connection.
When I dealt with V... in a different country a few years ago, they were awesome. The engineers were really clued in and helpful. Now though, in this country? Those engineers couldn't find their bum with both hands and a map. I'm told the quality of V... drastically varies from country to country.
no subject
Date: 2008-11-04 10:58 pm (UTC)no subject
Date: 2008-11-04 11:06 pm (UTC)--
"Memento Mori Ergo Carpe Diem"
no subject
Date: 2008-11-05 01:49 am (UTC)no subject
Date: 2008-11-05 12:33 am (UTC)no subject
Date: 2008-11-05 03:38 am (UTC)I had the unenviable position of being the semi-un-offical 'Mr fix-it' at the last-but-one place I worked at... I didn't answer calls. I browsed the ticket stack and looked for stuff that hadn't been followed up, followed it up, and kicked ass if needed, until what was broken got fixed, or we had a damn good reason why it hadn't.
Which is what one gets for being conscientious and doing a good job I guess... you get handed a worse one.
no subject
Date: 2008-11-05 02:54 pm (UTC)