Mass Hysteria

Today was the last day of school for all of the students, so obviously yesterday was the day that everything went wrong…

It began around 10:30am yesterday morning. I started getting calls that e-mail wasn’t working. It had been working earlier, so I just assumed that we were being flooded with viruses from the outside again (it seems like this happens weekly).

A while later I realized that I couldn’t even get into the server over the network. So I drove to the high school to check it out. Once I got there, I realized the C: drive on the mail server was full. Windows isn’t smart enough to rotate log files automatically, so I had to delete a few hundred megs of log files. It was almost lunch time by now, and I figured everything would be fine, so I went to lunch.

When I got back to my office, the server still wasn’t working right. So I tried the next logical thing with a Windows server, I rebooted. After the server rebooted I recieved a few e-mails from earlier in the day. I called a few people and let them know that we were back in business.

Then, about 5 minutes later, everything went down – the mail server was gone, the internet was down, big problems. So I drove back to the high school. The problem was obvious as soon as I opened the door to the server closet. Instead of the deafening roar of a thousand cooling fans, I was greeted with absolute silence. The power was off.

Everything in our server closet was plugged into one circuit and it was overloaded. Some guys from our maintenance department came over and reset the breaker, but it immediately tripped again. So I started borrowing extension cords and power strips and trying to get things moved over to outlets on different circuits.

In the end, I split everthing between three different circuits and got things going again. This involved an extension cord strung accross the floor of the library, and a few power strips plugged into power strips to reach the other side of the server closet, but at least we were operational.

Thankfully, we were able to find an electrician who could help us out right away. They put in four new dedicated circuits in the server closet yesterday. The were finished by around 5:30pm so I was able to get everything moved back around and ready to go for today.

Everything was smooth sailing today and we should be in good shape for a while now. Oddly enough, I don’t think the students even knew there was a problem. They all got out at 12:30 yesterday and things were back to normal today.

It’s days like this that make me miss Dr Pepper…