We have a mission critical server. It's a terminal server, and acts as an application server for internal work and remote users. About 6 months ago there was a "glitch" in the raid controller and it stopped mirroring the drives. So now it was running on one SATA drive, and the other was doing nothing. I kept saying this was unreliable at minimum -and compounding this was the refusal to back it up to our tape system. OK so that's the situation.
My coworker yesterday was tasked with adding 4 gig of ram into the system, performing a backup, and reying to get the RAID mirroring working. I wanted to do it over the weekend when there would be less of a time crunch but anyway... He installed the memory: All was OK. He tried to make a ghost image from a bootable CD, but it wouldn't see the drive. He instead figured that he would just use some sort of windows backup freeware utility to back up the drive. He made a backup. (I personally do not trust Windows environment backup utils, and would have stopped and put the drive into another computer and ghosted it)
He says he went into the RAID bios setup and selected mirroring. The system ended up formatting the drive and mirroring the blank drive -everything was gone *poof*
The backup made through the Win backup program was corrupt. All gone. We recovered the files off the formatted drive BUT it took all day to extract those files and put them onto the network.
Our TS is still down this AM. This could have been avoided had they listened to me and made regular images of this drive. It's very frustrating working for a place that has little concern for file security on mission critical systems.
is the other guy new?
Rule 1: Never...I mean always verifiy your backups before doing something with anything.
Wow thats pretty shitty. I have full backups run every night and the tapes are taken to a fireproof lockbox in our bank. They are on a 30 day rotation.
High costs initially but the benefit is great. There was some old database corrupt nobody noticed for several weeks but we were able to still go back and restore in no time.
I feel your pain.. no one listens to at work either although I am right 99.9% of the time....
Man John that blows. Times like that i am really glad that we have nerf guns in the office. That way we can vent in a safe non-throwing stuff way. A good ranting never hurts either. Glad to hear that you were able to recover the files and that the server is on its way back to functioning.
I cannot even *imagine* losing the data from a server at work.... ???
I have accidentally deleted a reporting database a couple of times. :laugh: Stems from having to run an old dos based report which I have setup on my laptop, which has 2 instances MSSQL installed and about 5 or 6 other servers reg.ed in Enterprise Mangler too. Just switching between the 2 servers and forget which one I was currently accessing and not looking.
On another note, all my servers are setup RAID 5 which gives me both fault tolerance and a boost in speed. Any specific reason you are using mirroring other than maybe the server only has 2 drives?
I feel your pain! My favorite, still unto this very day... Keep in mind this is before optical mice became the standard
User: My mouse is acting weird.
Me: What is it doing?
User: Can you bring me a new one?
This would be the THIRD mouse this month. I decided the shot-gun re-image/replace method was no longer hacking it. So I walk down there without a mouse, knowing she was experiencing a severe case of PEBKAC (Problem Exists Between Keyboard And Chair)
User: Where's my new mouse?
I open the bottom of her current mouse and find a wad of her hair. (She was losing her hair LOLOLOL I laugh cause I am too.) I drop the hair in her trash can.
Me: Just open it up and clean it out whenever it acts up.
She stood in total silence. For a brief moment, making me think that she might be either embarassed or comprehending what I said. Then she goes ballistic, demanding a new mouse.
Me: This mouse if fine, it will work the same as before just try it.
User: I want a new mouse.
We go for a couple rounds of this.
Me: I'm not giving you a new mouse until at least 3 months from now unless this one comes back in a million little pieces.
The user proceedes to smash the mouse and then jump on its' remains. I return a half hour later with an ancient logitech TRACKBALL. She used it.
Me: If you break this one I'll find something worse, I promise.
I had no problems after that.
Remember the wise words of Bones:
"I'm an IT not a babysitter!"
Dude, my little sister knows how to properly ghost a drive and to verify backups.
Quote from: Cannon Fodder on June 08, 2006, 05:44:12 AM
On another note, all my servers are setup RAID 5 which gives me both fault tolerance and a boost in speed. Any specific reason you are using mirroring other than maybe the server only has 2 drives?
The new NAS I'm putting together Monday has 4 drives & will be set to RAID 5. Most of our servers are dual opteron servers with 2 SATA drives. Mirroring would be fine IF the damn things were actually backed up occasionally.
ANd yes, I understand that my coworker screwed up HUGE.
-1st mistake was ignoring the fact Ghost wouldn't even see the drive.
-Second mistake was using a freeware/trial windows driven backup program instead of pulling the drive and ghosting it on a desktop with SATA, then investigating why the controller is goofed up.
-Third mistake was not verifying the backup even succeeded with crummy Windows freeware backup program.
-Fourth mistake was not even sending an email telling people TS puked -that pissed of a lot of people.
-Fifth mistake was after recovering almost everything off the formatted drive to a another file derver, he decided to recover it again from drive to drive and then repair the Win install with Serv 2003 disc -the second recovery was not as complete as the first and lots of files are missing. I kept saying using the gigabit switch would make it go pretty fast AND the file recovery was 99% complete there.
-Sixth mistake (in my mind) was that he didn't just say "I screwed up." Instead, he blamed the SATA controller for the problem.
-seventh mistake was not physically restoring TS users mydocuments and desktop folders. Some users had nothing after his restore.
-eighth mistake is not backing up the drive now that it works (mostly) and most apps have been reinstalled.
I swear some IT people like to fly by the seat of their pants. Not me. I'm Mr. Cautious.
Quote from: john on June 08, 2006, 09:53:16 PM
I swear some IT people like to fly by the seat of their pants. Not me. I'm Mr. Cautious.
Sometimes i think this is a biproduct of people going into IT work cause they thought that is were all the money is, so basically for the wrong reasons. I have seen what you described many times and many different jobs. Its sad really, i am with you, i would rather take more time and make sure that everything is there before blowing it away, but some have faith in stuff that any joe schmoe can download and use and don't feel they have to check. Those people cause more migrains than anyone... Way to get it back up again though John.
Quote from: john on June 08, 2006, 09:53:16 PM
ANd yes, I understand that my coworker screwed up HUGE. ...
I swear some IT people like to fly by the seat of their pants. Not me. I'm Mr. Cautious.
And how much did they say your upcoming raise is going to be!? :icon_mrgreen:
Seeing they look at us as equals...
Not much.
Wow, if you want me to take your co-worker out back and work him over with a baseball bat just let me know.
You in AZ? I just might take you up on it.
You have no idea how much I flipped out when I was given a new list of directives from the owners about how the IT support staff is to handle work on the servers. I won't say the details, but lets just say I ended my rant with "should I just sit in the corner and jerk off all day?" He makes one big mistake and we both get punished. I have never taken out a server.
Reactionary decisions are seldom the most prudent.
Well as soon as my battery comes in I'll ride down for ya.
http://maps.google.com/maps?f=q&hl=en&q=reno+to+phoenix&ll=36.332828,-115.070801&spn=6.441577,10.305176&om=1
Whaaat!?! Would you seriously take I-5? :o
I guess there's something to be said for traveling through populated areas.... :icon_rolleyes: But for 890 miles? When you could have most of a 738 mile trip all to yourself? (Yerlington, Hawthorne, Tonopah, Beatty, Las Vegas, Hoover Dam, Kingman, Phoenix)
If that looks too hot, you could stay "on top" for a while as you ride past Yosemite. Then comes the tough decision: Continue past Sequoia and "chicken out" through Ridgecrest and Palm Springs for 770 miles (and miss Hoover Dam) or take a left through Death Valley for a total of 748. I don't think I could pass it up ... I doubt it's any warmer than Phoenix most of the time. :)
I just google mapped it.
Just giving you a hard time. :kiss3:
Streets & Trips came up with the same route when I plugged the 2 endpoints in. I guess it must be the fastest of all possible routes (assuming strict adherence to posted speed limits)....
This may be something for another thread, but I'm interested in what route you think you really would take, and why. (If you don't mind my asking.) I've crossed Nevada from side to side on both I-80 and on US 50, but I haven't done any North-South stuff and would like feedback from those who have.
http://educationseek.com/images/us-map-edu.gif
Just follow the outer perimiter at first and then criss cross through it.