I smell opportunity...
Mar. 11th, 2008 10:58 pm![[identity profile]](https://www.dreamwidth.org/img/silk/identity/openid.png){kind=small}
siliconshaman.livejournal.com posting in ![[community profile]](https://www.dreamwidth.org/img/silk/identity/community.png){kind=small}
techrecoveryI shouldn't exploit the failings of fellow tech's... but sometimes you gotta wonder what monkeys are they employing?
So... End of shift, Monday. My first day after training. [ie, death by powerpoint presentation]
It being the beginning of the month the Bills have hit the Doormats for $MegaUtilityCompany which I have the dubious honor of being a lowly CSA for. [hey, it's a paying gig, and times are tight] And it's holy hell o the floor because $MUC just hiked the prices and announced a whopping big profit at the same time.
Half an hour before end of shift...and a deathly hush suddenly descends as the computers all lock up. Not a BSOD event, but network wide it all just goes into Guru mode, and of course nothing can be done because we have these spiffy new phones that don't have buttons, just a length of Cat6 connecting them to the Boxen.
"
Hmm... thinks I, and on a hunch checked the network clock, sure enough, Windows had done it's usual nasty trick of resetting to default GMT, ie, not DayLight savings when updated. [yeah, we're in Britain] Which meant that the servers were running backups half an hour before end of shift, not half an hour afterwards... and thus a tad busy to actually do anything else.
Floor managers reaction when I point this out;
"
*headdesk*
Methinks I may be moving sideways into the I.T dept at earliest opportunity. I suspect there may be an opening there soon, if not several. Besides, I could use the extra bananas!
So... End of shift, Monday. My first day after training. [ie, death by powerpoint presentation]
It being the beginning of the month the Bills have hit the Doormats for $MegaUtilityCompany which I have the dubious honor of being a lowly CSA for. [hey, it's a paying gig, and times are tight] And it's holy hell o the floor because $MUC just hiked the prices and announced a whopping big profit at the same time.
Half an hour before end of shift...and a deathly hush suddenly descends as the computers all lock up. Not a BSOD event, but network wide it all just goes into Guru mode, and of course nothing can be done because we have these spiffy new phones that don't have buttons, just a length of Cat6 connecting them to the Boxen.
"
Oh yeah
" says the floor manager, "it's been doing that about this time of day ever since they updated the net.
Might as well pack up and go home."Hmm... thinks I, and on a hunch checked the network clock, sure enough, Windows had done it's usual nasty trick of resetting to default GMT, ie, not DayLight savings when updated. [yeah, we're in Britain] Which meant that the servers were running backups half an hour before end of shift, not half an hour afterwards... and thus a tad busy to actually do anything else.
Floor managers reaction when I point this out;
"
Ohhh, I'll mention that to the I.T department! They've been trying to fix this for weeks!
"*headdesk*
Methinks I may be moving sideways into the I.T dept at earliest opportunity. I suspect there may be an opening there soon, if not several. Besides, I could use the extra bananas!
![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
no subject
Date: 2008-03-11 11:48 pm (UTC)I have to admit that I am fighting the Timezone fight with my personal laptop.
For some reason XP has decided that it likes being in Germany i.e. where I bought the laptop. Alas, I live in China. So I changed the timezone. Problem fixed? MWuahhah. NO. In random intervals, after a couple of rebootings, XP decides it wants to be German again. Incredibly annoying.
no subject
Date: 2008-03-12 12:19 am (UTC)no subject
Date: 2008-03-12 12:26 am (UTC)no subject
Date: 2008-03-12 10:54 am (UTC)no subject
Date: 2008-03-12 11:44 am (UTC)I spent 4 years in non-IT roles. I fixed more IT problems than the IT people did in a few of them. The unfortunatly thing is this doesn't typically mean you will be moved to IT, just that your office staff will begin using you instead of them.
no subject
Date: 2008-03-14 06:44 pm (UTC)(1) Run production DB/files/etc on 3 way RAID mirror (technically a 3 way mirror of RAID 5 arrays, but that is moot here)
(2) At 3AM, stop the webservers (the daemon, not the boxen) and the DB.
(3) Split off one of the 3 mirrors (remember, DB is dropped, so in consistent state)
(4) Spin everything back up.
(5) Back up the split off mirror to tape.
(6) Rejoin it back into the pool when complete.
Total downtime ends up being about 30 seconds, and the load imparted by shoving the split off mirror to tape adds like nothing. The machine has 10CPU's, so it's not easily killed by a simple backup.
It's not actually that simple - there are other things which happen in that process (compress, encrypt and send to offsite colo backup, replicate to another server internally, etc.) but you get the general idea...