Greets, and my tale of woe
Jul. 13th, 2007 10:18 pm![[identity profile]](https://www.dreamwidth.org/img/silk/identity/openid.png){kind=small}
roo2.livejournal.com posting in ![[community profile]](https://www.dreamwidth.org/img/silk/identity/community.png){kind=small}
techrecoveryHello all. My name is Roo, and I'm a tech support minion. I also go by the title of Software Ninja (I'm in ur 'puter, installin' ur softwarez), as well as a few others. I do departmental PC support in a university, and there are times when my life is a living hell.
So, we got pwned over spring break. Someone hacked into several computers and used them for serving music and movie files. We found things like eMule, Limewire and uTorrent on the infected PCs. It was not fun. We first got the heads-up when the network security guys called us and told us one of our computers was scanning the network and there was major P2P traffic. Being the minion, I was dispatched to pick up the 'puter and haul it off. Our university has a computer security group and they wanted to confiscate the drives to do forensics. To be honest, I think they got some shiny new forensics tools and we just happened to be their first guinea pigs, but whatever. Our stated policy is that users are responsible for backing up their data - important work-related stuff resides on the server, and we won't back up your iTunes library for you. We'll show you how to do backups on DVD, CD, USB key or whatever, but you're responsible for making sure that your files are safe. My boss is very firm on this because there were apparently Incidents in the past that made him and the department chair believe that this was the best policy. At any rate, I wasn't too worried about data on the drive in question because of this. Even better, the computer belonged to one of our staff photographers (I'll call her Photographer A). Since we make money off the photographers, we have processes in place and redundancies to make sure the important images are safe. When they insert a CF card into the reader, they use a small java applet that gives the files their proper names and copies them to both the local hard drive and the server (which is backed up regularly). Plus, every week or so the photographers burn their images to DVD and file them in binders based upon date. All the critical work related data is safe. Y'all can stop laughing hysterically now.
So, I hauled the computer off. The security guys removed the drives for analysis, and I tried to deal with some of the other disasters we had going on. The next morning, Photographer A called me, asking why her computer hadn't been returned. I explained that the security team had the drives and so I couldn't do anything. I offered to hook her up with a temporary computer, but she said she'd brought her laptop and would use it. No biggie. Until she called back that afternoon, asking about her computer. I repeated that the security team had it and it was out of my hands. She spent about 20 minutes plaintively saying "But how am I supposed to get my work done?" I again offered her a loaner, she again refused. Repeat this. Every day. For two weeks. I know that's a long time, but the entire university was dealing with attacks and intruder activity at this point, and most of us techies were scurrying around like mad, subsisting on caffeine, pizza, sugar and fried stuff. I eventually had to get my manager to call her manager so she'd quit badgering me. This is when the ugly heart of the matter was exposed - she hadn't been using the java applet because it was 'too slow'. And she hadn't been backing up her hard drive onto DVD either, I have no clue why. In short, the only copies we had of her pictures were on that hard drive, which was sitting on some security guy's desk, and which would not be returned to us if they found social security numbers or sensitive information on it since it might be used as evidence in criminal proceedings. There were 'splosions, since we had customers waiting on images that were on that drive.
We finally got her drive back. I backed the images up carefully, scanned them just to be extra paranoid, formatted the drive and did a clean install and had it back to her, complete with restored images pretty darn quickly. One would think that would be the last time we faced a disaster like that, right? Right?
We got a call on Monday - another computer scanning the network, P2P activity. The security guys have become more blase these days, and said if there's no sensitive data on it then don't worry about sending the drive over to them - they're backed up enough as it is, especially now that it's summer. I was dispatched to take care of it. I unplugged the computer from the network, deleted the remote access tools and the accounts that had been created, deleted Limewire and did some general cleaning. We have equipment coming in right now and I'm buried, and the user leaves for vacation this Monday, so we agreed he'd be able to use his computer this week as long as it stayed off the network. I'd haul it off and clean it up and have it waiting for him when he got back. He agreed. Oh, and this is another photographer - I'll call him Photographer B. My boss asked me to also grab a third photographer's machine (I'll cleverly call him Photographer C) and rebuild it as well. Photographer C installs lots of crap on his system and it's really unstable - every time it boots it comes up with errors. It drives me nuts. But because of departmental politics he's been given admin on his machine, so there's not much we can do about it. Anyway, the bottom line is I'm to scrub the machines clean and rebuild them. After the debacle over Spring Break, we're confident this group has learned their lesson. Only they haven't. They each have about 250GB of image data on their drives that doesn't exist anywhere else. Not on the server, not on DVD, nowhere but on their hard drives. They want me to bring over an external drive and back up the drives for them, and they want to be there and tell me which files get backed up and what their file names should be (since they didn't use the applet that gives them proper names). Since this sounds like my definition of hell, and since the biggest external drive I have is only 300GB and it has data from other computers that are being rebuilt, I go to my boss, who manages to borrow a 500GB external drive, which he tells them they can figure out themselves. Somehow, they're unable to do this and insist that their hands must be held during this process. Boss tells them to pound salt, since we're in the middle of assigning new IPs to every device in our department (I can't believe my university still uses static IPs) and we're focusing on that project. They somehow figure out the really complicated backup process on their own.
This morning we went to their building to change IPs. I wasn't around, but my boss reportedly hit the roof. You see, Photographer B had plugged his computer back into the network. He claimed he had no idea how that happened, but somehow I doubt the cleaning crew did it. So my boss confiscated his ethernet cable. And for good measure, he told me to yank the card (which I couldn't, seeing as it was on the mobo). While we're dealing with that, these two photopgraphers are whining about how long it takes to burn backups and how they want faster burners (the deadline for new equipment orders was 30 days ago) and listing all the problems they're having with their computers that they want fixed ASAP because they just can't get their work done with these issues (like intermittent sound problems - real relevant when editing images, guys). I finally ended up pointing out that both of their computers will be getting rebuilt next week, which should take care of the persnickety problems they're complaining about. My boss pretty much threw up his hands and stalked off to spend a couple of hours in the gym to cool off. He sometimes has anger control issues, and usually avoids having anything to do with the photographers. I totally understand why.
Anyway, most of my users really are great. But this particular bunch of rotten apples is why I'm looking for a new position (that and I'm woefully underpaid, especially for the crap they throw at me).
That's the end of my tale of woe. I just had to share it.
Days like this make me wish I was a drinker.
So, we got pwned over spring break. Someone hacked into several computers and used them for serving music and movie files. We found things like eMule, Limewire and uTorrent on the infected PCs. It was not fun. We first got the heads-up when the network security guys called us and told us one of our computers was scanning the network and there was major P2P traffic. Being the minion, I was dispatched to pick up the 'puter and haul it off. Our university has a computer security group and they wanted to confiscate the drives to do forensics. To be honest, I think they got some shiny new forensics tools and we just happened to be their first guinea pigs, but whatever. Our stated policy is that users are responsible for backing up their data - important work-related stuff resides on the server, and we won't back up your iTunes library for you. We'll show you how to do backups on DVD, CD, USB key or whatever, but you're responsible for making sure that your files are safe. My boss is very firm on this because there were apparently Incidents in the past that made him and the department chair believe that this was the best policy. At any rate, I wasn't too worried about data on the drive in question because of this. Even better, the computer belonged to one of our staff photographers (I'll call her Photographer A). Since we make money off the photographers, we have processes in place and redundancies to make sure the important images are safe. When they insert a CF card into the reader, they use a small java applet that gives the files their proper names and copies them to both the local hard drive and the server (which is backed up regularly). Plus, every week or so the photographers burn their images to DVD and file them in binders based upon date. All the critical work related data is safe. Y'all can stop laughing hysterically now.
So, I hauled the computer off. The security guys removed the drives for analysis, and I tried to deal with some of the other disasters we had going on. The next morning, Photographer A called me, asking why her computer hadn't been returned. I explained that the security team had the drives and so I couldn't do anything. I offered to hook her up with a temporary computer, but she said she'd brought her laptop and would use it. No biggie. Until she called back that afternoon, asking about her computer. I repeated that the security team had it and it was out of my hands. She spent about 20 minutes plaintively saying "But how am I supposed to get my work done?" I again offered her a loaner, she again refused. Repeat this. Every day. For two weeks. I know that's a long time, but the entire university was dealing with attacks and intruder activity at this point, and most of us techies were scurrying around like mad, subsisting on caffeine, pizza, sugar and fried stuff. I eventually had to get my manager to call her manager so she'd quit badgering me. This is when the ugly heart of the matter was exposed - she hadn't been using the java applet because it was 'too slow'. And she hadn't been backing up her hard drive onto DVD either, I have no clue why. In short, the only copies we had of her pictures were on that hard drive, which was sitting on some security guy's desk, and which would not be returned to us if they found social security numbers or sensitive information on it since it might be used as evidence in criminal proceedings. There were 'splosions, since we had customers waiting on images that were on that drive.
We finally got her drive back. I backed the images up carefully, scanned them just to be extra paranoid, formatted the drive and did a clean install and had it back to her, complete with restored images pretty darn quickly. One would think that would be the last time we faced a disaster like that, right? Right?
We got a call on Monday - another computer scanning the network, P2P activity. The security guys have become more blase these days, and said if there's no sensitive data on it then don't worry about sending the drive over to them - they're backed up enough as it is, especially now that it's summer. I was dispatched to take care of it. I unplugged the computer from the network, deleted the remote access tools and the accounts that had been created, deleted Limewire and did some general cleaning. We have equipment coming in right now and I'm buried, and the user leaves for vacation this Monday, so we agreed he'd be able to use his computer this week as long as it stayed off the network. I'd haul it off and clean it up and have it waiting for him when he got back. He agreed. Oh, and this is another photographer - I'll call him Photographer B. My boss asked me to also grab a third photographer's machine (I'll cleverly call him Photographer C) and rebuild it as well. Photographer C installs lots of crap on his system and it's really unstable - every time it boots it comes up with errors. It drives me nuts. But because of departmental politics he's been given admin on his machine, so there's not much we can do about it. Anyway, the bottom line is I'm to scrub the machines clean and rebuild them. After the debacle over Spring Break, we're confident this group has learned their lesson. Only they haven't. They each have about 250GB of image data on their drives that doesn't exist anywhere else. Not on the server, not on DVD, nowhere but on their hard drives. They want me to bring over an external drive and back up the drives for them, and they want to be there and tell me which files get backed up and what their file names should be (since they didn't use the applet that gives them proper names). Since this sounds like my definition of hell, and since the biggest external drive I have is only 300GB and it has data from other computers that are being rebuilt, I go to my boss, who manages to borrow a 500GB external drive, which he tells them they can figure out themselves. Somehow, they're unable to do this and insist that their hands must be held during this process. Boss tells them to pound salt, since we're in the middle of assigning new IPs to every device in our department (I can't believe my university still uses static IPs) and we're focusing on that project. They somehow figure out the really complicated backup process on their own.
This morning we went to their building to change IPs. I wasn't around, but my boss reportedly hit the roof. You see, Photographer B had plugged his computer back into the network. He claimed he had no idea how that happened, but somehow I doubt the cleaning crew did it. So my boss confiscated his ethernet cable. And for good measure, he told me to yank the card (which I couldn't, seeing as it was on the mobo). While we're dealing with that, these two photopgraphers are whining about how long it takes to burn backups and how they want faster burners (the deadline for new equipment orders was 30 days ago) and listing all the problems they're having with their computers that they want fixed ASAP because they just can't get their work done with these issues (like intermittent sound problems - real relevant when editing images, guys). I finally ended up pointing out that both of their computers will be getting rebuilt next week, which should take care of the persnickety problems they're complaining about. My boss pretty much threw up his hands and stalked off to spend a couple of hours in the gym to cool off. He sometimes has anger control issues, and usually avoids having anything to do with the photographers. I totally understand why.
Anyway, most of my users really are great. But this particular bunch of rotten apples is why I'm looking for a new position (that and I'm woefully underpaid, especially for the crap they throw at me).
That's the end of my tale of woe. I just had to share it.
Days like this make me wish I was a drinker.
![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
no subject
Date: 2007-07-14 04:45 am (UTC)no subject
Date: 2007-07-15 12:20 am (UTC)no subject
Date: 2007-07-14 04:51 am (UTC)Start hard, that way you get it over with.
no subject
Date: 2007-07-15 12:21 am (UTC)no subject
Date: 2007-07-14 12:20 pm (UTC)Noooooo problems! I went over and unplugged it and left an email (CC'd to various folks) informing him that there would be no more problem with people running illicit software off CDs...
no subject
Date: 2007-07-15 12:22 am (UTC)no subject
Date: 2007-07-15 02:26 am (UTC)no subject
Date: 2007-07-14 01:14 pm (UTC)Our local policy is that, well, desktop PCs are expendable. After all, they might catch fire, get stolen, blow their disks, get fried by a heavy electrical surge or have peanut butter fed into their PSU by a 3-year-old.*
[* All of these have happened, except the last one. So far as I know..]
So, we tell users that any vital data they have needs to be stored in a safe place. We handily provide large network fileservers with multiply-redundant (online- and offline-) backups to help in this regard.
Then, if their desktop does blow up, we just swap in a spare, freshly auto-installed box and they can pick up where they left off.
At least, that's the idea. In practice, when I ask "Is there any vital data on this machine?" the answer is occasionally "yes".
I have this stange quirk of actually liking users, so unless we're completely overloaded I do try to get their data back - even if it's just personal stuff. Unless the HDD is completely fried, most of it is normally recoverable.
But we're only required to make a best-effort - if the data's gone, then it's their fault for not storing it safely.
no subject
Date: 2007-07-15 12:25 am (UTC)no subject
Date: 2007-07-14 09:58 pm (UTC)no subject
Date: 2007-07-15 12:26 am (UTC)no subject
Date: 2007-07-15 02:17 pm (UTC)Of course, it's always fun to point out that they've been using the software for a week and the problem started after they started deleting random files in /usr/lib, so maybe it isn't my software update after all?