I've got a stumper of one here.

[jecook]

And I'll cut it just for you.



I've been fighting with my company's Ghost Console for the past week, and I need some help, because I've got what I think is a stumper. It's got me stumped, at least.

The issue is that we have an office that has a lab of 13 machines that we re-image remotely on a regular basis (or did until it stopped working earlier this week). The server that runs the show is at a different office some 5 miles distant across a Metro Ethernet / DS3 connection.

The server is at 10.10.128.x, the machines are at 10.20.10.x. Pings, network traffic goes just peachy. Also, the remote client install goes fine as well. What happens is that the server will initate a task (either image creation or clone) and it will sit and time out. The clients appear to just sit there and do absolutely zilch. It's like they don't catch the signal to start the task.

I was able to initally duplicate the problem on a single test machine back at the facility where the server lives, but I thought I had it fixed by completely removing the machine from the console, removing the tasks it was associated with, and then re-adding it from scratch (including reinstalling the client). This appeared to work just peachy, but I can't get the machines I've removed to show back up in the console.

Anyone know of a way to either manually add machines to the console, or to force it to re-discover all the machines on the network?


::puts on flame-resistant suit and digs out the whips::


EDIT:::: It looks like a multicast over WAN, combined with the machines not wanting to report back to the server across said WAN link, possibly using the same multi-cast method. I'd asked one of the BNAFH* to look at it, and we'll see what happens.

*- Bastard Network Admins From Hell. DOn't piss them off, or they will route all your traffic through SimonNet. :)

Comments

[jon787.livejournal.com]

Will you cut your wrist just for me too?

/me dodges the whip

[toxico.livejournal.com]

Shoot the hostage?

[jon787.livejournal.com]

No Jack, poor people are crazy. I'm eccentric.

[bornofchaos.livejournal.com]

I understood 1 word in 4. But I send you productive-karma anyway.

I'm an enthusiastic follower of "Plug the whole frigging lot out, go for a pint, come back and hope for the best" school of thought.

[gilmoure.livejournal.com]

Yes, the go-for-a-pint school or fixun' things is my favorite as well.

[kizayaen.livejournal.com]

Flame resistant suit?

I know I sure don't have the balls to flame a mod.

Now if you were a first-time posting n00b, that'd be a different matter entirely...

[ghostdandp.livejournal.com]

No idea what the problem is, but for troubleshooting purposes I'd toss a packet snifer somewhere on the network and see what's going back and forth.

[hisamishness.livejournal.com]

not a bad place to start

Do you have a security group that deals with firewalls? Did they do any updates or patching or "general maintenance" recently? You'd be surprised how many times routers go south after extended uptime because someone forgot to write the running code to the boot flash....

[jecook]

I strongly suspect that either the network admins or the telco did something, but the network admins swear up and down that they did nothing.

Frankly, I'm thinking that it's the telco and their metro ethernet thing-y causing the multicast issue. I think I'll toss the thing into forced unicast mode just for kicks to see what it does.

[jecook]

Well, one would expect a mod of a group to follow their own rules at least... :)

[kizayaen.livejournal.com]

I've seen a fair number of exceptions handed out for people who are judged to have put in their time and contributed to the community for a good while. I should hope you'd qualify.

[jcaswell.livejournal.com]

the network admins swear up and down that they did nothing.

Hah! I'll lay money they're lying to you. Give it a couple of weeks and they'll say "oh, we did change this, but that won't affect it!". Never trust a network admin who denies everything :)

[kuang.livejournal.com]

Do you have WINS in place by any chance? If the discovery process is failing whenever it tries do do $stuff because of multicast problems, Ghost should have a go at using that instead.

According to Symantec, if clients don't respond it'll try a second time and them mark the machine record as timed out before moving on - this can take up to 3 min per attempt. What they don't say is how to clear the timed out status, and I haven't used it in a while so I'm buggered if I can remember :)

Just a silly thought (because you have to have at least one per comment ;) ). Did one of the last images to successfully go out have any tweaks over the previous version that may stop the client running? I only ask because of a mare of a time I had with SMS2003 whereby the clients had set their cache size back to the default without asking and a rights cockup had locked off the temmporary storage location, so nothing would deliver.

[dysan27.livejournal.com]

Or to actual, non-RTFM, problems. Though the 'been here for a while' does count for much.

[anamacha.livejournal.com]

perhaps you forget the ever-popular method of percussive maintenance?

[gilmoure.livejournal.com]

[anamacha.livejournal.com]

*chortle* he had more than one?

[redqueenmeg.livejournal.com]

Sure, you can't forget "2 Legit!"

[anamacha.livejournal.com]

I needed to remember that?

[redqueenmeg.livejournal.com]

That song rocks. Reminds me of that two-week period when everyone was wearing white pants and navy jackets. "We crush the strong and percolate the weak..." hee

[susano-otter.livejournal.com]

Telnet across the port that Ghost uses.

If the telnet works, your problem isn't network-related; talk to the software vendor about why their app is misbehaving.

If the telnet fails, your networking team or the telco is lying to you, and there's been a recent network change; deliver richly-deserved beatings until the problem is solved.

[dirtymatt.livejournal.com]

Make sure multicast traffic can get over the link. I'm not certain, but I think Ghost Console won't work if multicast traffic is being blocked.

Here Here!

[ace-brickman.livejournal.com]

Was there a new router/switch in-between those 5-mile links lately? If multicast traffic is blocked, Console won't be able to talk to the clients for any ghosting sessions. However, other application data (NetBIOS names of said clients showing up in the console i.e. "finding" them in the console) is allowed. Boggled me too when I ran into this problem.

[ace-brickman.livejournal.com]

I had the same problem about 5 months ago.. Sorry for the late comment. Did the solution present itself?

[jecook]

Not yet, sadly.

See edit for what I've at least narrowed it down to.