Stupid provider games...

[greenmansgrove.livejournal.com]

Two different issues, only related by the fact that they indicate the general level of customer service available from larger companies these days. Oh, and related by the fact that they're pissing me off!


I have a client who's got a Netgear Prosafe switch. Get a call from them telling me that they have no internet connectivity, and no connectivity to the server. Go onsite to troubleshoot, and find that the gigabit port that the server is plugged into is dead. Server also provides DNS and DHCP, which explains they had no internet connectivity. Move the server to a functioning gigabit port, and contact Netgear to get a warranty replacement, since they have a limited lifetime warranty for these switchs. Proceed to jump through many flaming hoops of bad phone tree, getting disconnected time after time. Eventually, call the corporate office and say "why the hell can't I get ahold of someone in tech support without getting disconnected?" and get a technician. Explain the situation, who says "Oh, I can issue an RMA and send out a new switch." Problem solved, right?

The Netgear switch that I eventually got Netgear to replace under warranty? It blew out two ports in two days, both about a week after the switch was installed. Now I need to get back on the phone with Netgear. So I do. And the game starts ALL OVER AGAIN! This time at least I didn't get hung up on. This time, I didn't actually get to talk to anyone who could help me, but got the opportunity to leave a voicemail on their pro-series tech support line, asking for a return call. I made this call on Thursday. Oddly enough... I haven't heard back.
(sarcasm)And I've got all the time in the world to sit on the damn phone trying to fight my way through their phone tree.(/sarcasm)
Might be better to tell the client that they should just pay for a new switch... from a different company.

In other news, some script kiddie tried to hack one of my client sites last night, using a brute force password attack against the administrator, guest, and info accounts. The administrator account has a very secure password, the guest account is disabled, and there IS no info account, so he had no luck. But when I tried to contact XO Communications to get the attack stopped at the router level, I was told by the technician on duty that he couldn't do that, and that I would have to talk to someone in the fraud or abuse departments. Which were (of course) closed at 11pm when I was speaking to the tech. He opened a trouble ticket for next level support and told me that they would have to call me back, which of course, they never did.

So, since I couldn't get the ISP to do anything, I stepped in with a bad and temporary fix. Behind the XO router, we have a small (home model) Netgear router, primarily used for the purpose of running NAT. I reconfigured the router so that it wouldn't talk to the XO router, blocking the attacks. Of course, this also killed internet connectivity for the client, but since it was at midnight, I didn't think that would be a significant issue. I went in early this morning (as soon as they opened) and re-configured the Netgear router to talk to the XO router again. I started checking logs to see if the attack was still ongoing, but it looks like the script kiddie gave up sometime over night.

Now I'm on the phone with XO, getting bounced from department to department as they try and figure out which group should be handling this. So far, I've been to the DS1 repair group and the Abuse group. The DS1 repair group supervisor has been trying to help, and tried to get me to the Operations Support group, but apparently has no clue as to the inner workings of his company, and neither he nor the Abuse rep (who I spoke to in place of the Operations Support ret, and who doesn't actually have access to the routers) have been able to help me thus far. The Abuse rep mentioned the Network Security group, but apparently neither one of them have any contact information for that group (probably for "security" reasons). So the supervisor that I've been working with is now trying to contact someone he knows who "can do all this stuff. He's not at work,but let me call him on his cell phone."

And that's where I'm at so far today. Instead of being at another client site entirely, configuring the workstation that I was originally scheduled to set up. Fortunately, they're being very understanding, and have told me not to kill myself to get there.

GAAAAAHHHH!!!! THIS is precisely why we don't recommend XO as an Internet Service Provider.

Comments

[medusa-nw.livejournal.com]

Argh, don't even get me started on those bloody Netgear switches! We've now replaced all of ours, no prizes for guessing who the manufacturer is this time...

This is news?

[ptomblin-lj.livejournal.com]

I get about 2 or 3 ssh bruteforce password guessing attempts a day, both on my home box and on my colo box. In both places, the following takes care of it for me:

! /bin/sh
#
# ssh-bruteforce
#
# Author: Michael Greb <michael@thegrebs.com>.
#
# Version: @(#)ssh-bruteforce 1.0 26-Mar-2005
#

set -e

PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
DESC="iptables for ssh brute force mitigation"
NAME=ssh-bruteforce
SCRIPTNAME=/etc/init.d/$NAME


#
# Function that starts the daemon/service.
#
d_start() {
iptables -N SSH_WHITELIST
iptables -A SSH_WHITELIST -s 192.168.1.0/24 -m recent --remove --name SSH -j ACCEPT
iptables -A SSH_WHITELIST -s 69.207.172.43 -m recent --remove --name SSH -j ACCEPT
iptables -A SSH_WHITELIST -s 74.202.84.133 -m recent --remove --name SSH -j ACCEPT
iptables -A SSH_WHITELIST -s 74.202.84.134 -m recent --remove --name SSH -j ACCEPT
iptables -A SSH_WHITELIST -s 129.21.192.213 -m recent --remove --name SSH -j ACCEPT
iptables -A INPUT -p tcp --dport 21:22 -m state --state NEW -m recent --set --name SSH
iptables -A INPUT -p tcp --dport 21:22 -m state --state NEW -j SSH_WHITELIST
iptables -A INPUT -p tcp --dport 21:22 -m state --state NEW -m recent --update \
--seconds 60 --hitcount 4 --rttl --name SSH -j LOG --log-prefix SSH_brute_force
iptables -A INPUT -p tcp --dport 21:22 -m state --state NEW -m recent --update \
--seconds 60 --hitcount 4 --rttl --name SSH -j DROP
}

d_stop() {
iptables -F
iptables -X SSH_WHITELIST
}

case "$1" in
start)
echo -n "Starting $DESC: $NAME"
d_start
echo "."
;;
stop)
echo -n "Stopping $DESC: $NAME"
d_stop
echo "."
;;
*)
# echo "Usage: $SCRIPTNAME {start}" >&2
echo "Usage: $SCRIPTNAME {start}" >&2
exit 1
;;
esac

exit 0



[greenmansgrove.livejournal.com]

Netgear used to provide good solid hardware and good support (when it was needed). I hate it when a vendor like that goes bad.

[sdaemon.livejournal.com]

my experiences with XO have never been good. Their easy-to-use account administration page is anything but. Their top-tier support guys seem to be savvy, but wading through the challenged simians and marketeers at the bottom tiers makes me want to jam screwdrivers into my eyes.

that said, they're also not the worst ISP I've had to deal with. I think DeltaCom still takes the cake there, with CBeyond putting in a strong showing as well.

[medusa-nw.livejournal.com]

used to indeed. No longer.

Re: This is news?

[mattcaron.livejournal.com]

The following should work if you don't have iptables installed:

route add -host ATTACKING_IP_ADDRESS gw 127.0.0.1

So, your machine will never return any packets to the attacker..

Apparently, this works too:

route add -host ATTACKING_IP_ADDRESS reject

But I have not tried it.

Of course, a firewall is better, but if you don't have one, it is better than what the original poster did.

[jimbojones.livejournal.com]

The ProSafe routers are still about the best thing going in the small/medium market. When did the switches start sucking? I've got six or seven in the field (full gigabit 24 port and 48-port with 4 gigabit ports) that have done fine for a year or so.

[jimbojones.livejournal.com]

Deltacom is fucking AWFUL. I actually had a scriptmonkey (read: tier 1 "support") person tell me that SOP was that if I needed a technician who could actually shell into the router, that would be a minimum 24-hour wait for a callback.

Customer had no internets at the time, too.

AWFUL.

[sdaemon.livejournal.com]

In CBeyond's defense, some of their problems are equipment based rather than personnel based. One of my clients uses CBeyond's managed VPN service. I needed some changes made to the DHCP options (I think it was DNS servers being pointed to) for one of the VPN sites. I would call CBeyond, they would connect to their VPN router (that was also handling DHCP) and make the change. Shortly later I'd get a call that the internet was broken at that site. It took a few repetitions of this to find exactly what was going on, but apparently changing the DNS server entries in DHCP on the VPN router was also resetting the default gateway. And DHCP reservations were right out. I don't know what kind of effed up proprietary hardware they were using, but it definitely caused quite a headache for me and the CBeyond engineers helping me. They were as perplexed by the behavior of their equipment as I was.

Also, in defense of both DeltaCom and CBeyond (and all other crappy ISPs), there's only so much they can do when they're forced to run on top of Bellsouth/AT&T's lines. Sometimes, the problem really is the physical line, and endless fingerpointing will ensue.

[jimbojones.livejournal.com]

I've worked for a CLEC; I know the biz pretty well. There's nothing about Bell's copper that makes a CLEC unable to connect you to an actual technician, as opposed to a glorified secretary with an Excel spreadsheet in front of her, for 24 hours plus.

[medusa-nw.livejournal.com]

All in the last year. We've replaced at least half a dozen.

[jimbojones.livejournal.com]

SIGH... guess it's back to the generics next time I need a big switch, then.

Half a dozen failed out of how many deployed, if I might ask?

[medusa-nw.livejournal.com]

I think we've got three more out there now. We have a couple on standby for those! :-)

[jimbojones.livejournal.com]

ouch... 70% failure or so then?

I had similar results with the Seagate Barracuda 200GB drives. The 160s were fine, the 400s were fine, but every... single... 200GB I ever deployed developed MAJOR media issues within about 18 months. Every last one.

[medusa-nw.livejournal.com]

Yup, pretty much.
Before the Netgear switches we had a similar problem with 3Com apparently. That was before my time though.

[sdaemon.livejournal.com]

true dat.

Re: This is news?

[jecook]

Stupid iptables n00b question:

I see how the script does it's thing, but how would one use it in a real life scenario?

I ask because, well, I'd like to try and park it on my unix machine, which sees quite a number of brute force attempts. (so far, I've had exactly one incident where they managed to get in, but that was a weak password issue.)

[jecook]

Yep Yep.

Re: This is news?

[ptomblin-lj.livejournal.com]

Simple. I put it in /etc/init.d, and put in the appropriate symlinks in the other /etc/rcN.d so that it's started on boot up, and then I sit back and wait for entries in /var/log/messages that mention "SSH_brute_force".

Re: This is news?

[syberghost.livejournal.com]

My solution was even simpler:

PermitRootLogin = without-password

Another good one is "forced-commands-only" and of course the ever-popular "no".

Re: This is news?

[ptomblin-lj.livejournal.com]

Well, duh. Surely in this day and age, nobody allows root to login with a password. But the password guessers try hundreds of different account names. They probably figure that once they get in they can escalate privs. I just find it simpler to cut them off before they've hit my ssh server hundreds of times in a few minutes.

One of my two colo boxes has "PasswordAuthentication no" and "UsePAM no" because I'm the only person who logs on, but the other one has several user accounts on it, and so I have to allow password authentication.

Re: This is news?

[pauamma]

I take it that this place doesn't have a rule against UI?