(no subject)

[daddykatt.livejournal.com]

Welcome to @Hosting_Company - Where our motto is "It doesn't have to be right, it just has to work."

This weeks misadventures include our entire 3rd floor datacenter having the AC finally die, and over a thousand servers bite it from heat exhaustion within an hour.

After that, we visit our intrepid graveyard worker while he attempts to reason with the Chinese gold farmer

And we wrap our show up with mis-adventures in hardware scavenging.

And for spice, let's also throw in some co-worker-stupid.

-----

Our 3rd floor DC houses about 1600 or so machines... most of them are Intel Atom boxes that are packed so tightly together that they really get NO airflow, so they run nice and toasty to begin with. Add in that we also have loads of mid-tower cases, with minimal fans in them too, and you have a recipe for an oven. Or disaster, that works too.

Over the last few weeks, the temp has been on a steady climb, and we have had near daily drive/system failures because of heat. And then one of the 2 and a half AC units shut down. (We have 2 17ton coolers and 1 portable AC unit on that floor) Of course it had to be one of the BIG coolers too. pretty much every Atom had to be shut down while we had the unit fixed, but that doesnt solve the fact that no airflow reaches those racks to cool them. So they build the ducts up on both of the big units to blow OVER TOP of all of the racks... doing nothing to direct the airflow INTO the racks to clear out the 90+ degree air that surrounds all of the machines.

On the plus side, the air around the AC's is no longer around 95 degrees... it is now a balmy 84.

Fun stuff that.

...

The other day I have a client open a ticket because his server is down. Now, we d not give priority to any kind of issue that is a single customer problem... we go by the queue for the most part... reboots get bumped up because they are fast. I had a customer open a ticket in english, explain (badly) what was going on enough for me to work on his box (which was hosed and in need of a reinstall) so I ask him for approval to reinstall... and he answers back in Chinese... full on character script and all... I tell him that I can't understand him, and need him to use American. He responds back again... in Chinese...

I ask him AGAIN to use american, and finally he tries to sell me WoW gold to expedite his fix...

...

Hardware... We are a datacenter, and a fairly active one at that, one would think that we should order things like, oh, say DRIVES or memory, or even boards and raid cards... So with these things missing, we must scavenge from all of the un-used servers... or give our customers major free upgrades (an Intel Atom comes standard with a 200 gig drive, but we dont have any of those, so they get 500's... but we are out of those too, and 1T drives... so they get 1.5T drives for FREE)

And now the bosses are bitching bec ause we are out of hardware and that we are scavenging from servers that we SHOULD keep around for a few weeks in case the customer who canceled their server wants it back again.

Whoops

...

And THEN we get to the coworkers...

See, we are a big bunch of geeks, and that causes problems... To quote my boss "You guys built an Auto-PXE system that TALKS to you, but you cant all agree on an OS to use."

We have guys here that use windows, I use a HackinT0sh, one guy uses slackware, another uses fedora, our boss uses OpenBSD, another guy uses mandriva, we have 2 guys that use ubuntu, and a guy that uses debian 99% from only the command line.

Which raises an issue with workstations... we have 4 workstations in the main area, and 1 on our 3rd floor... with constantly changing OS'es. Bosses are pissed, and threatening to make us all use Windows 2000.

I think that's all I have for today.

Comments

[the-reda.livejournal.com]

/he tries to sell me WoW gold to expedite his fix.../

Omg, giggling so hard here. I really hope that was not one of ours.

[onyxrising.livejournal.com]

You know, that's far better than what some customers have offered my techs to get things done.
... then again, my techs would probably take the WoW gold quietly and never say anything.

[natalie-i-am.livejournal.com]

Which raises an issue with workstations... we have 4 workstations in the main area, and 1 on our 3rd floor... with constantly changing OS'es. Bosses are pissed, and threatening to make us all use Windows 2000.
^^ This is "cruel and unusual punishment"

[mouser.livejournal.com]

Somewhere I have a copy of Win95...

[dave-iii.livejournal.com]

I've actually got a copy of 3.1 laying around somewhere... Last time I installed it (an experiment prompted by a bad case of upgrade-itis) it ran slick as snot, but couldn't actually DO diddly.

[ghostdandp.livejournal.com]

Still have a copy of Windows... well it's be Version 1, but it's just called windows. It was pretty much a bad shell, and I can't install it on anything since I can't find a working 5.25" disk drive.

[captpackrat.livejournal.com]

I've still got a copy of Microsoft Bob.

(Hey, it was a great interface... if enjoy The Sims.)

[bothunter.livejournal.com]

I have a copy of DOS 6.22 still in the shrinkwrap. I'm tempted to open it so I can request a copy on 5.25" floppies.

[tuba-man.livejournal.com]

You can still get 3.11 from MSDN subscriber access. Apparently some developer somewhere still checks that box when he does his surveys?

[siliconshaman.livejournal.com]

Hey, at least they're not being threatened with Vista..

[daddykatt.livejournal.com]

Vista is still capable of doing what we need to function. And it has MORE ability to function that 2K in this environment.

And we have an ISO for 3.1 lying around, but the only reason they dont threaten that is because it doesn't support what we HAVE to have for work

[vulpisfoxfire.livejournal.com]

Not really...they need to learn the lesson 'Your personal machines are one thing, but quit reinstalling the communal workstations'. I'd make them use ME.

[metao.livejournal.com]

DURPA DURRRR I SPEAK AMERICAN

[vulpisfoxfire.livejournal.com]

Yeah, that was the one annoying thing in this other giggle-worthy entry. I'd be tempted to use either British or Austrailian English just to annoy the OP. :-)

[daddykatt.livejournal.com]

Nah, I specified American, because I am just that, not English, and I don't speak (the queen's)English. English is a good language, American is a bastardization of English that is in common use in the states.

[ravan.livejournal.com]

On the OS issue - WTF? IMO, you get a pick from 3 - 1) a Windows version compatible with admining your windows boxen, 2) A Mac version compatible with the hardware you have, or 3) a Linux version that matches what you run on most of your servers. Your desktop you can customize with your login. Every tech needs to know how to use all three - even if they hate the Windows.

This is why they won't hire me - I'm too much of a logical bitch about stuff like that.

[daddykatt.livejournal.com]

Yeah... one of our guys "Slackware Dude" almost refuses to even load or touch our customer boxes that have windows on it... he apparently used to work for MS and such is his hate for the product.

[ravan.livejournal.com]

I'm almost that bad - I got very abused while supporting both Linux and Windows by arrogant, entitled Windows users. They froze out my Linux time with their constant Windows shit. I now get very rage filled if I have to do anything with Outlook, Exchange, AD, or finance people who don't know how to use fucking Excel.