Tech Support HELL

Worst case scenario, maybe. My guess would be they'll call Symantec first and inquire why BackupExec screwed the pooch. Then, when Symantec says to call MS, they'll do so.

We Don't Just Think Different, We Think Backwards™

From:

In what fashion did the backup fail? BackupExec likes to report a completely failed backup when it's just a message or two in exchange store that are corrupt and couldn't be opened, so in that case, you can still recover everything else...

From:

What I'm being told is that the backups never succeeded from day one, yet for some reason that I cannot comprehend it was left installed to attempt to operate and fail every time.

The policy on backups here, or any type of contingency plan really, is less than what one would call "existant." There's one Oracle server that, if it ever fails, pretty much makes us unable to do business because it's never been backed up. Not once. Oh, and someone made the decision to migrate it to SQL recently too. Fun fun.

From:

Story so far is that one of the drives in the array has shuffled off this mortal coil, and the rebuild failed for the same reason. Upon a successful rebuild of the array everything was present save for the Exchange logs.

How to prevent it in the future? Don't buy the refurbished hard drives from Woot.com. :P

From:

Yeah, but there are different grades of success and failure. If it's from day one, then I assume all the hardware is functional. After that, it can be a security problem of BE not having logon credentials to access to exchange store, agent misconfiguration (or lack thereof), or plain dirty database. In the latter case, you'll recover most of the data, but in any of the former... makes me glad I'm an ocean and a sea away from that clusterfuck. Among the most important things I check for on a new server installation is how does backup perform, and set it up to email notifications after every job run, with a full log attached. Failed backups are a fairly common occurance, mostly people not replacing tapes on time, or tapes and/or drives wearing out (especially common in 5-8 year old systems), but having backups not work from day one is a catastrophe.

From:

Were the logs on a separate partition that failed to survive the rebuild? Nothng else makes much sense...

From:

Update on that: The log files were there, except E00.log. So, about 4 hours of mail lost once it finishes rebuilding the database.

Weird thing was, the failure was on a SAN with a RAID 6 array. We don't know why it didn't just keep on truckin' minus the one drive.

From:

Turns out the initial suspicion of BE being the culprit was incorrect, as an inexplicable POST error from a drive in the array made the whole array shit itself. Everything but the E00.log was present once the rebuild succeeded and the databases are being rebuilt now. In RAID 6 one would think that the array would keep functioning in the event of one drive failing, but that didn't happen for whatever reason.

There will essentially be a four hour window from 8PM to midnight where any mail will be lost, plus the lost productivity from today. We're drawing up proposals for a backup system that works, including mail archiving and another SAN to mirror the primary. Clustering is also under consideration.

From:

Finally got a chance to talk with the guys working on the server directly... Our guess is that the currently-in-use log file was in the process of being modified when the drive crapped out, so it was corrupted. It's really the only thing that didn't survive but of course that initiates a ripple effect...

From:

I can't help it...

Well then take a nap. THEN FIRE ZE MISSILES!

*cough* Yeah.

From:

Actually, I wondered how long it'd take.

From:

I've been dealing with the Lusers on my end, and I finally couldn't take it any more.

From:

See, there aren't a lot of Lusers to deal with here on a regular basis. Not much to do except answer the occasional phone call and play WoW.

I'm just highly embarrassed that we have been relegated down to the level of Ill-Prepared Luser. Guess we learned the hard way.

From:

I just may have to put you on my flist if you play WoW and are doing techie stuff. :P (Yeah, this is my priest.)

From:

You're welcome to; just be aware that I'm usually rather crude, random or both in my journal.

*looks at yer icons* Guess that's not an issue. Carry on then.

I haven't posted any WoW-related stuff lately due to a personal situation (that you'll learn aaaall about) and a soon-to-be-corrected heat issue with the new system I put together taking away most of my play time. I'm about to start playing regularly again though. Verily, geekery will ensue.

From:

*snicker* Yeah, crude and random are completely what my journal is about. But of course, I've not posted anything in my journal in a while. Prolly ned to get on that.

From:

Nice to have the budget to do that kinda thing :)

From: