Saturday evening saw a rough stormfront bludgeon its way through Atlanta. On its way through, it disrupted power and phones at my company’s corporate office. Our network communications were temporarily disconnected, but the UPS batteries on the servers did their jobs and the servers stayed online through the outage and were humming along just dandy when power and network came back up.
Then, at 11:30pm-ish, someone went to the electrical room in the back of the building and manually pulled all of the main breakers. See that slot machine looking arm on the right side of the image? Six of those had been pulled down, disconnecting the circuit. This killed power to the entire building. I didn’t know power had actually been restored at any point so I figured it was a power company problem and that they’d just get things working again. After I fielded more than 100 phone calls from our store locations I decided to call the power company myself. Marietta Power informed me that I was the only one to report an outage in the area (which is mixed residential and commercial). They recommended I go to the office and check the breakers. So I drove 45 minutes to arrive at the office at 1pm. It’s hot and dark. The server room must have some serious UPS batteries b/c the big servers were still on and running. It was 110 degrees in the room, according to the mercury thermometer we keep in there. Now, I didn’t know the electrical room existed so I had to call the power company to come out and fix the power. It took them an hour to get here (with me sitting in the hot dark the entire time) and they found the problem immediately. I went back to where they showed me the problem and I flipped the power back on. Back in the building we had lights and more importantly AC. Sweet AC! It’s now about 3:30 and I’m making sure all the servers in the server room are OK. The SQL Server is beeping at me something horrible and a 30 minute call to Dell later (to fix the external RAID arrays) and the beeping stops and my brain stops bleeding. During the hour I was working in the server room the temperature fell from a scorching 110 degrees to a balmy 98 degrees. I spent another hour or so stomping around making sure everything I could was up and running (and it was during this last hour of my time at the office that two of my co-workers showed up to help out) before leaving at 5:20 to try to make it across town for dinner at 5:30. We pushed dinner back 30 minutes and the Asahi beer and hibachi food both tasted great
By the time the weekend was over, I’d been in the office on Sunday for 4.5 hours and had fielded something like 200 phone calls. Needless to say, we’re already implementing a phone tree to alert all remote locations in a more efficient manner should something like this happen again.