Why can't stuff break at a normal friggin hour?
 

God dammit.

Tonight has been a nightmare. Well, not really a nightmare. For that I'd have to be sleeping :(

Had a disk to replace tonight in an SSA attached to an old ass R/S 6k. I'm new to this stuff so you're probably as confused with the acronyms as I am. I get by, you will too. So anyway, we go out to the computer room, open the box with our shiny new disk and....it's the wrong part. IBM shipped the wrong damn part. So the CE orders up the new disk. It's local (as in here in Sac), so no biggie (I've had parts fail at 3am before and have to come in from Hayward. By car. IBM apparently has no helicopters, rocket ships or dudes with jet packs to ferry parts.). It's 11:30pm when we order the disk. The people on the ordering side are apparently playing pong or something and decide not to order the disk for like 2 hours. We get the disk at 3am :mad:. We replace the disk, go to start the cluster and start seeing some odd errors. What's that? The power supply failed! Well, luckily there's two power supplies so the box is up, but if one is down and the other fails, bad news! So the CE just left to go grab the PS to come back and replace it. He's been here since 7pm, me since 8pm. He probably won't be back till near 6 and then who knows how long it will take to swap it :(

The only benefit to being up this late is that I'll probably get home about the same time as the woman and we'll make breakfast and have a few drinks together. It's ok to drink at 6am if you haven't gone to sleep yet.

So to make this more of a discussion than a rant...

Don't you guys hate it when stuff fails at the most inopportune times? Tell me all about it and I'll read it tonight when I wake up!

yes.

Yes... For the most part, my company has given up, and has entire Fault tolerant clusters and farms with on-site spares for key systems, and are progressing to full geographically dispersed High Availability (Dual active where possible)

I know many can't afford that level of Fault Tollerance, but you'd be surprised how inexpensive on-site spares are compared to the cost of 4 hour support over time.

OMG, that's sucks dude...I've always wondered why things tend to fail when no body's around.

I used to work as a nighttime dispatcher for my old trucking company, coordinating trucks all over CA, NV and AZ, and I swear to God, every night I'd have to find people to fix shit broken down along the road or in the most remote locations...It's hard to deal with mechanics on the phone, let alone at 3 AM.

Enjoy your day sleep.

We have a lot of similar high availability tech, unfortunately we have so much different hardware we couldn't really keep spares for everything :(. Especially on these boxes that are so old.

Normally I get a break from 1am to 5am, which I use to sleep, but the other night our Delta flight was running 3 hours behind. We didn't finish until 3:30, I had an hour and a half to sleep before the morning flights went out.

Its Harsh