Information Technology – Under Water

I’ve yet to write anything here concerning the thing that consumes most of my time, IT. I can’t think of much that is more dry or uninteresting to talk about than IT, so I don’t.

I love the challenge of it, all the little puzzles and the problem solving; it engages a huge part of my mind. I learn new things every day and it earns me a living—but IT as the stuff of inspiration or conversation—not so much. My boss has to drag me to IT expos, even though they give away free stuff like foam dice, keyring flashlights, and steak dinners…  None the less, I guess it is about time for some kind of piece about the thing I spend so much of my life doing… So here goes.

Several years ago I was working as IT Director, Chief Cook, and Bottle Washer for a very vigorously growing company in downtown Manhattan.

I was very happy at this job for the most part and my boss was more than good to me. Change then came in the form of business strategies and management structure. My Mother and Father both became ill. Life changed and it was clear my own time for change had arrived.

I have to say that while I lack the nice office and view; and a good percentage of the salary; I feel better about things now—working for an organization that functions not for profit but to make human lives better in areas of war, terror, and natural disaster. Despite my personal pay check I know the company I’m supporting has a much deeper purpose and I feel very good about that.

With all that said I wanted to tell an IT horror tale from my days at that other company.

I was sitting in my office, next door to the server room, gazing out the window at the Chrysler Building and across the East River to Queens; when in walks Eddie the maintenance man. Eddie was looking quite concerned and strode right into my office and began checking all the pipes under the window and above the drop ceiling.

“May I help you?” I asked…

“Oh, I’m just trying to see if I can find a small leak… You haven’t seen anything have you?”

“No, I’m afraid I haven’t – can I give you a hand with that file cabinet over there?”

“No, I’ve got it, thanks…”

“Well, ok then.. I’m going to go to the men’s room and leave you here at it Eddie, ok?”

“Yeah, no problem…”

So, I leave and go to the men’s room and take a seat upon the john. I call the third stall from the right “the library” because someone always leaves at least one newspaper in there. Usually its the Wall Street Journal, but occasionally you’ll find the Post (much more appropriate for the surroundings) or even a Vanity Fair if you’re lucky.

I stayed a leisurely fifteen minutes or so before leaving the men’s room to walk back to my office. As I turned the corner, I saw all the girls from accounting standing at the elevator. We all seemed surprised to see each other.

Someone asked me what did I think of it all?

“What do I think of what…?”

“You know, the FLOOD.”

“The WHAT, oh I know – you guys are pulling my leg aren’t you?”

“No, really… You mean you haven’t seen it?”

“No, I was reading…”

At this point they all kind of looked at each other like ‘Oh Shit’ and I hurriedly made my way back to my office and the server room.

As I rounded the corner I beheld a sight I shall not forget for the rest of my life. Hundreds of gallons of water were pouring a deluge directly over one of four server racks. The water was pounding the top of the rack and cascading down the sides. There was already four inches of water on the floor.

Our developer and two man IT team stared on in shock and wondered what to do about the ELECTRICITY, seeing as the power supply to the servers was fed with around 400 Volts. Sparks and smoke were flying…

I decided that something had to be done so I put out my elbow and ran as fast and hard as I could run through the water toward the master power switch, which I successfully hit (with my elbow) into the off position. The only remaining power issue would be the existing batteries and breakers in the power supply which were still live to the breakout panel feeding the servers. We later manually pulled out batteries and switched off breakers on the back of the power supply itself. We had no documentation on it and none of us had been around when it was installed. We did the best we could with it and got APC on the phone promptly thereafter to do a completely safe shutdown.

The fire department came, axes ready but there wasn’t much for them to do at that point.

The Vice President of Global Operations came downstairs and asked me what was our plan. I more or less told him I didn’t know. We both stood and surveyed the destruction for a while as it slowly sunk in that about 95% of our IT infrastructure and data had been completely wiped out.

We began pouring water out of servers and taking them apart to dry, with hope that something might power on and give us enough time to recover something. Of course we had data on tape stored off site, but that was two weeks old and required retrieval. Also, our tape drives and library were part of the wreckage – so at that particular moment tape wasn’t going to do us much good.

I was able to get one domain controller operating and seize the master active directory roles onto this server. I was then able to get one mail server and data store on line. So before the night was out I was able to restore email which was a pretty big accomplishment.

We didn’t have much hardware that wasn’t ruined. Most of our older hardware was less damaged – the cases were bigger, the internal components spread out more, and the fans were not placed directly next to dense circuit boards. The newer servers were all one or two rack space jobs and the internal electronics were really densely packed. The fans were set in such a way that they just sprayed water all over everything. None of those new servers survived, but some of the drives did.

It became clear I was going to need some working servers to try and bring everything back on line. Of course this was a Friday night, and we were now in the middle of a snow storm as well. I was able to get an entry level Compaq server and a high end Sony Vaio workstation from a local IT dealer. This was the best I could do. The store was about two avenues over and 8 blocks up. I went over and purchased the two machines with my Amex. I also purchased power strips, extension cords and anything else I could think of. Somebody else went to a department store and picked up some hair dryers and towels. The hair dryers brought my cell phone back to life.

We placed all of this equipment (server, workstation, cords, miscellaneous hardware) onto a cart and two guys from the store and I wheeled it back in the snow.

Over the next 36 hours (I worked 36 hours straight) I was able to bring all of the data back on line in a temporary fashion using the purchased server and workstation, plus some of the other equipment that was still partially running. Over the coming weeks and months we turned the experience into a positive one by replacing all our old hardware with brand new hardware using the insurance money.

I received a very generous bonus which I promptly turned around and used to buy my first digital SLR.

It was quite an experience and I really hope I never have to go through anything quite like that again!

Leave a Comment