Tech Support HELL

Worst case scenario, maybe. My guess would be they'll call Symantec first and inquire why BackupExec screwed the pooch. Then, when Symantec says to call MS, they'll do so.

We Don't Just Think Different, We Think Backwards™

(deleted comment)

From:

Story so far is that one of the drives in the array has shuffled off this mortal coil, and the rebuild failed for the same reason. Upon a successful rebuild of the array everything was present save for the Exchange logs.

How to prevent it in the future? Don't buy the refurbished hard drives from Woot.com. :P

From:

Were the logs on a separate partition that failed to survive the rebuild? Nothng else makes much sense...

From:

Finally got a chance to talk with the guys working on the server directly... Our guess is that the currently-in-use log file was in the process of being modified when the drive crapped out, so it was corrupted. It's really the only thing that didn't survive but of course that initiates a ripple effect...

(deleted comment)

From:

Update on that: The log files were there, except E00.log. So, about 4 hours of mail lost once it finishes rebuilding the database.

Weird thing was, the failure was on a SAN with a RAID 6 array. We don't know why it didn't just keep on truckin' minus the one drive.

From:

pixelchik.livejournal.com

I personally like to sacrifice a chicken to the Exchange gods when I pull the /p(ray) switch... :D

From:

canray.livejournal.com

Voodun tech support. Don't laugh, it works. Noone knows why, but it works.

Honestly, we should have Pointy Hats as a requirement to be Computer Techs just like Wizards (And Wizzards) do on Discworld.

Personally, I'd hate to have to troubleshoot Hex. ;-P

"Every time we move the FTB away from Hex, it comes up with the error: +++ Whaaaaaaaaa! MINE! +++. Any ideas?" "Don't take away the FTB unit."

From:

In what fashion did the backup fail? BackupExec likes to report a completely failed backup when it's just a message or two in exchange store that are corrupt and couldn't be opened, so in that case, you can still recover everything else...

From:

What I'm being told is that the backups never succeeded from day one, yet for some reason that I cannot comprehend it was left installed to attempt to operate and fail every time.

The policy on backups here, or any type of contingency plan really, is less than what one would call "existant." There's one Oracle server that, if it ever fails, pretty much makes us unable to do business because it's never been backed up. Not once. Oh, and someone made the decision to migrate it to SQL recently too. Fun fun.

From:

Yeah, but there are different grades of success and failure. If it's from day one, then I assume all the hardware is functional. After that, it can be a security problem of BE not having logon credentials to access to exchange store, agent misconfiguration (or lack thereof), or plain dirty database. In the latter case, you'll recover most of the data, but in any of the former... makes me glad I'm an ocean and a sea away from that clusterfuck. Among the most important things I check for on a new server installation is how does backup perform, and set it up to email notifications after every job run, with a full log attached. Failed backups are a fairly common occurance, mostly people not replacing tapes on time, or tapes and/or drives wearing out (especially common in 5-8 year old systems), but having backups not work from day one is a catastrophe.

From:

Turns out the initial suspicion of BE being the culprit was incorrect, as an inexplicable POST error from a drive in the array made the whole array shit itself. Everything but the E00.log was present once the rebuild succeeded and the databases are being rebuilt now. In RAID 6 one would think that the array would keep functioning in the event of one drive failing, but that didn't happen for whatever reason.

There will essentially be a four hour window from 8PM to midnight where any mail will be lost, plus the lost productivity from today. We're drawing up proposals for a backup system that works, including mail archiving and another SAN to mirror the primary. Clustering is also under consideration.

From:

Nice to have the budget to do that kinda thing :)

From:

I think the realization that it should have been done from the beginning is sinking in to the minds of the Ones Who Write the Checks. The least expensive option will likely be chosen, but we're laying all the options out like good, thorough drones.

From:

I can't help it...

Well then take a nap. THEN FIRE ZE MISSILES!

*cough* Yeah.

From:

Actually, I wondered how long it'd take.

From:

I've been dealing with the Lusers on my end, and I finally couldn't take it any more.

From:

See, there aren't a lot of Lusers to deal with here on a regular basis. Not much to do except answer the occasional phone call and play WoW.

I'm just highly embarrassed that we have been relegated down to the level of Ill-Prepared Luser. Guess we learned the hard way.

From:

I just may have to put you on my flist if you play WoW and are doing techie stuff. :P (Yeah, this is my priest.)

From:

You're welcome to; just be aware that I'm usually rather crude, random or both in my journal.

*looks at yer icons* Guess that's not an issue. Carry on then.

I haven't posted any WoW-related stuff lately due to a personal situation (that you'll learn aaaall about) and a soon-to-be-corrected heat issue with the new system I put together taking away most of my play time. I'm about to start playing regularly again though. Verily, geekery will ensue.

From: