Insert Head A into Desk B

[kalidor.livejournal.com]

Please note the following. Dates, names, and ips have been changed to protect the innocent, and the pathetically stupid.

Client : Midsized corperate with several offices. Ticket deals with newest and busiest location. Probably one of our top five clients.
ISP : Three person company
Our Responsiblity: The Computers, Lan. Connection to WAN is grey area due to bosses hazy partnership with ISP.

Layout : Computers --- Linksys VPN router --- Netopia T1 router ------ ISP T1 router @ NOC


10/29
Arrived on site to replace Netopia T1 endpoint that customer dropped. Installed device in ten minutes. Called ISP for settings as D-mark was not labeled with setting information. Spent 2.75 hours on phone with ISP going through various settings as they had not seen that Netopia type before and did not know what the proper settings were for it to Connect to their T1 line.
Ticket Closed.


11/01
Ticket Reopened. Customer complaining about extreme latency and periodic disconnects. Arrived on site and found latency. Unfragmented continuous ping to Linksys DSL router proved reliable connection. Unfragmented continuous ping to Netopia router proved reliable connection. Unfragmented continous ping to ISP's T1 router proved choppy but acceptable connection. Ping to backbone IP (4.2.2.2) proved horrible connection past ISP's NOC. Attempted to phone ISP but got static. On a hunch, tested various MTU's. Administrativly throttled Linksys MTU to 1200 and connection seems to have stabalized. Suggest ISP debug their NOC. Ticket Closed - SEP.

11/03
Ticket Reopened. Customer is experiencing frequent timeouts. ISP claims they can ping with 1500 bytes to Linksys, from their off NOC computers. Linksys does not respond on ICMP, administrativly. ISP suggested that Linksys is dying due to faulty hardware. Replaced device with test linksys. Issues continued. Connected laptop directly to Netopia Router. Issues remained. Contacted ISP while looking at configurations. ISP asked me to look at Quickview on the Netopia and finding no errors, said it was our problem. Still have good pings on Lan and DMZ. T1 pings choppy, NOC pings are crud. Checking logs in linksys found Netbios attack from Netherlands being blocked by linksys. (ISP is allowing NetBIOS?) Found logging function on Netopia. Logs indicate massive framing and PPP problems on T1. Emailed logs to ISP. ISP claims if the T1 was truly having problems they would be seeing the same logs on their T1 endpoint. When asked if they new where the logs were, they said they could see the quickview well enough.Implemented NetBIOS blocking on Netopia.

11/5: 1300-1430
Went on site regarding continued internet problems. Was informed that ISP had diagnosed Virii on LAN side. Full scan of all three computers revealed no viral infections. Netopia logs were wiped. Linksys logs revealed continued attacks on NETBIOS port from Europe and the Caribean. Implemented harsh firewall on Netopia. Internet and VPN stabalized.

11/5: 1600-1700
Returned on site due to lack of internet connectivity. VPN Program still worked fine. Could find no issues. Perhaps DNS was timing out. At approximately 1647 the internet returned to normal with no configuration changes.

11/6 : 1130-1230
Returned due to lack of Connection. Netopia and DSL were being flooded by Netbios requests. Implemented Aggressive connection keeping on Linksys to compensate for T1 instabilities. Continued testing connection. Seemed fine after 45 minutes.



Network Accessable up thru 11/8 - 0035


04/11/07 07:49:11 PPP: IPCP negotiated, session 1, rem: 66.194.143.137
04/11/07 07:49:11 PPP: NCP up, session 1, Channel 1
04/11/07 07:49:11 T1: Received alarm condition cleared
04/11/07 07:49:05 PPP: Channel 1 up
04/11/07 07:49:05 >>WAN: T1 1 activated at 1536 Kbps
04/11/07 07:49:05 T1: Line is available with acceptable quality
04/11/07 07:48:58 T1: Remote framing recovered - clearing yellow alarm
04/11/07 07:48:56 T1: Receiving yellow alarm (RAI)- remote loss of framing
04/11/07 07:48:55 T1: Received alarm condition cleared
04/11/07 07:47:22 T1: Line is unavailable, due to excessive errors
04/11/07 07:47:15 T1: Lost remote framing - sending yellow alarm (RAI)
04/11/07 07:47:13 >>WAN: T1 1 deactivated
04/11/07 07:47:13 PPP-1 down: No Sync
04/11/07 07:47:13 T1: Receiving blue alarm (AIS) - all ones signal
04/11/07 07:47:12 PPP: Channel 1 up
04/11/07 07:47:12 PPP-1 down: Data protocols all down
04/11/07 07:47:12 PPP: NCP up, session 1, Channel 1
04/11/07 07:47:09 PPP: Channel 1 up
04/11/07 07:47:09 PPP-1 down: Data protocols all down
04/11/07 07:47:09 PPP: NCP up, session 1, Channel 1
04/11/07 07:47:06 PPP: Channel 1 up
04/11/07 07:47:06 PPP-1 down: Data protocols all down
04/11/07 07:47:06 PPP: NCP up, session 1, Channel 1
04/11/07 07:47:03 PPP: Channel 1 up
04/11/07 07:47:03 PPP-1 down: Data protocols all down
04/11/07 07:47:03 PPP: NCP up, session 1, Channel 1
04/11/07 07:47:00 PPP: Channel 1 up
04/11/07 07:47:00 PPP-1 down: Data protocols all down
04/11/07 07:47:00 PPP: NCP up, session 1, Channel 1
04/11/07 07:46:57 PPP: Channel 1 up
04/11/07 07:46:57 PPP-1 down: Data protocols all down
04/11/07 07:46:57 PPP: NCP up, session 1, Channel 1
04/11/07 07:46:54 PPP: Channel 1 up
04/11/07 07:46:54 PPP-1 down: Data protocols all down
04/11/07 07:46:54 PPP: NCP up, session 1, Channel 1
04/11/07 07:46:51 PPP: Channel 1 up
04/11/07 07:46:51 PPP-1 down: Data protocols all down
04/11/07 07:46:51 PPP: NCP up, session 1, Channel 1
04/11/07 07:46:48 PPP: Channel 1 up
04/11/07 07:46:48 PPP-1 down: Data protocols all down
04/11/07 07:46:48 PPP: NCP up, session 1, Channel 1
04/11/07 07:46:45 PPP: Channel 1 up
04/11/07 07:46:45 PPP-1 down: Data protocols all down
04/11/07 07:46:45 PPP: NCP up, session 1, Channel 1
04/11/07 07:46:42 T1: Received alarm condition cleared
04/11/07 07:46:42 PPP: Channel 1 up
04/11/07 07:46:42 PPP-1 down: Data protocols all down
04/11/07 07:46:42 PPP: NCP up, session 1, Channel 1
04/11/07 07:46:36 PPP: Channel 1 up
04/11/07 07:46:36 >>WAN: T1 1 activated at 1536 Kbps
04/11/07 07:46:36 T1: Line is available with acceptable quality
04/11/07 07:46:29 T1: Remote framing recovered - clearing yellow alarm
04/11/07 07:46:27 T1: Receiving yellow alarm (RAI)- remote loss of framing
04/11/07 07:46:26 T1: Received alarm condition cleared
04/11/07 07:45:13 T1: Line is unavailable, due to excessive errors
04/11/07 07:45:07 T1: Lost remote framing - sending yellow alarm (RAI)
04/11/07 07:45:04 PPP: IPCP down, session 1
04/11/07 07:45:04 >>WAN: T1 1 deactivated
04/11/07 07:45:04 PPP-1 down: No Sync
04/11/07 07:45:04 T1: Receiving blue alarm (AIS) - all ones signal
04/11/06 12:20:29 PPP: IPCP negotiated, session 1, rem: 66.194.143.137
04/11/06 12:20:29 PPP: NCP up, session 1, Channel 1
04/11/06 12:20:29 PPP: Channel 1 up
04/11/06 12:20:29 PPP: IPCP down, session 1
04/11/06 12:20:29 PPP-1 down: Profile changed
04/11/06 12:04:22 PPP: IPCP negotiated, session 1, rem: 66.194.143.137
04/11/06 12:04:22 PPP: NCP up, session 1, Channel 1
04/11/06 12:04:21 T1: Received alarm condition cleared
04/11/06 12:04:07 PPP: Channel 1 up
04/11/06 12:04:07 >>WAN: T1 1 activated at 1536 Kbps
04/11/06 12:04:05 T1: Received alarm condition cleared
04/11/06 12:04:05 T1: Receiving yellow alarm (RAI)- remote loss of framing
04/11/06 12:04:04 T1: Receiving blue alarm (AIS) - all ones signal

Comments

[oddball42.livejournal.com]

i almost wish you had posted the isp so i could avoid them in the future...

netbios indeed.

[jecook]

Crapo on a crutch. What a bunch of total morons. Our local phone CO does a better job, and I try to avoid them. :)

What is the ISP using for their T-1 endpoint?

[rettiger.livejournal.com]

See, now here's a difference between you and me. The moment that idiot said "The logs don't matter", I'd have demanded to be pushed up a level of support. Or two. And if they didn't give it to me, I'd have gone to a supervisor. That's bullshit, you could have saved yourself several days worth of work if you'd have just gotten an actual engineer on the phone instead of a monkey reading a script.

[kyndig.livejournal.com]

LOL suggesting they debug the NOC was the funniest thing I've read all morning. I work for an ISP and wish we could debug our NOC!

[codedigital.livejournal.com]

I can answer this one for him:

The guy he was talking to on the phone was the freaking owner of the company!

[kalidor.livejournal.com]

I won't give out the mane of the company, but here are two rules you can use:

1.If the ISP has less then ten employees, you will probably want to look elsewhere.

2.If they can't diagnose their own equipment, run!