techrecovery | Email says it all.

names redacted so I don't seem like a dick. Also, oldest to newest

--------------------------------------------------------------------------------
From: Account Manager Guy
Sent: Thursday, August 30, 2007 3:52 AM
To: Account Manager Guy; Regional Sales Manager Guy; Random Other Guys
Cc: NEW SVP; NEW DIRECTOR
Subject: RE: *customer* Down
Importance: High

Case ID 16286. TierIII Engineer1 and TierIII Engineer2 are working on it. From case notes:

Sales Engineer didn't realize he had setup a temporary cluster with the first of the new nodes and didn't reset it. When he powered on the new nodes enough of its configuration was the same as the production cluster (backend IP's and networking) that the production cluster started copying settings from the new node.

status no longer shows any data

names of all the nodes have been changed to *cluster2* instead of *cluster1*

--------------------------------------------------------------------------------
From: NEW SVP
Sent: Thursday, August 30, 2007 6:32 AM
To:

omg_teh_funnay
Subject: FW: *customer* Down
Importance: High

OMG,

Were you following the node add procedure when this happened?

Thanks,

SVP

--------------------------------------------------------------------------------
From:

omg_teh_funnay
Sent: Thursday, August 30, 2007 7:49 AM
To: TierII Engineer
Subject: FW: *Customer* Down
Importance: High

SVP thought this was me, it's not... it's you. So if you wanna look into this and all, that'd be great.

--------------------------------------------------------------------------------

From: TierII Engineer
Sent: Thursday, August 30, 2007 8:11 AM
To: New SVP
Cc:

omg_teh_funnay
Subject: RE: *Customer* Down

SVP,

OMG wasn't the one following this case. Myself and TierIII Engineer2 were working on it last night. This issue has been cleared up, I should have had them setup a temporary cluster originally with the new nodes then do a reset to just be safe, but because Sales Engineer told me that he hadn't setup anything with the nodes we proceeded with booting it while attached to the IB switch. Before the new node even made it to any kind of a login, it began changing the configuration of the production cluster because it had the same ip addresses as the other nodes ib cards. Again we have cleared this up and the customer is back up and running.

Thank you,
TierII Engineer