*headdesk*

[coyoteden.livejournal.com]

I just got an urgent e-mail from a place I do support on the side for: Their backups haven't run for 4 months. They are just now realizing this.

Problem: the server had been "administratively paused" 4 months ago. Oddly enough, it also shows it had last been manually rebooted for maintenance 4 months ago.

Solution: Unpause, backups start fine, and for future references, CHECK YOUR DAMN LOGS ONCE IN A WHILE!

Oh... I just realized this also means no one has patched it for 4 months, probably because no one has bothered to approve any updates. Fsck.

Comments

[jecook]

Ouch.

When I was doing the 'drive around the city fixing crap' job, we considered a server that had a bad backup thrice in a row a priority one call. (Priority one meaning "drop whatever you were doing and send a warm body over here to look at the damn thing and figure out WTF is going on!" type of call.)

4 months? that's just stupid.

[torkell]

Could have been worse. They could have found this out only after one of the drives went "fsck this, I hate the lusers, I hate this server, and I hate the admin, so I'll randomly corrupt just about everything".

Been there, done that, got the t-shirt. My mirrored-with-xcopy data drive did *not* die, and is still working fine (except for going out-of-warranty next month, at which point it'll probably go the same way). It was the non-mirrored non-backed-up system drive. And pretty much overnight it died on me. I'd left the system it was in powered off for a week (was away), came back to find a few Bad Messages in the event log after a few hours usage. Decided to sleep on it and start shifting stuff in the morning, started it up out of hibernation, and a couple hours later it killed winlogin.exe, and a few hours after that DriveImage barfed with a hard read error. I managed to get the stuff off eventually (there was nothing crucial, but I just knew I would want the odd config file from it) by installing Win2k to the data drive and setting xcopy to run for a few hours. This was after finding that Knoppix and well-and-truely-fscked NTFS does not mix. Neither does Knoppix and floppy drives.


(oh, and guess what? The warranty had expired a few months before. So I didn't even get a new hdd out of it :(

[gholam.livejournal.com]

I've had worse earlier this week. New client's server has been operating for eighteen months with a failed hard drive in the primary RAID5 array, when another hard drive failed last weekend. They were running backups - sort of - with one full backup on two tapes, first of which was left in the server thursday evening, and the second inserted after the weekend end, sunday morning; the rest was incremental updates. Of course, the server crashed in the middle of full backup, and Exchange got hosed.