Early in my career, I had to administer a server blindfolded. As the first web developer, I did not have much interaction with the IT Manager at first. He was busy dealing with the rest of the software engineering company. I remember he came to my desk on the first day and told me, “Here’s your machine. If you [mess] it up, I’ll reformat your hard drive. So don’t do anything stupid.” He gave me an ip address and the root password to the company’s website: “You’re the new webmaster, dude.”

Things went well until one day when I somehow erased the default gateway setting. Then, my telnet connection dropped and I was unable to reconnect. So I went downstairs to access the console.

Only one problem. The company webserver was an UltraSparc workstation–apparently one of two that had been used by software engineers for testing. At some point, the video card had failed and was cannibalized. Or maybe they could only afford one card for the two servers. But what I found downstairs was a server with no video card, on a pile of other discarded equipment with no serial cables handy and no clue what to do. And the website is now down.

Fortunately, there was a keyboard. I power cycled, waited a few minutes, then typed in “root,” hit carriage return, and typed in the password. Then I used vi to edit /etc/defaultrouter, hit “d” a few times to delete whatever was in the file, typed in the correct ip address of the gateway, hit control-Z, control Z and typed reboot. The server came back up and we were able to telnet in again from my desktop upstairs.

Now of course this situation sounds ridiculous today. How did the machine get in that state to begin with? Why was there no serial cable? Who knows? But so often emergencies and crisis situations present in ways that are impossible to predict in advance but easy to blame in retrospect. You deal with the situation, the cards you are dealt, while you plan for the future.

The real irony is that we never did buy a video card. When that server was moved to a hosting facility, I configured it using a laptop and a serial cable.

We are so far past all that, right? Well, sort of. Nobody is going to cheap out on video cards, but there is always something unexpected waiting to happen or not happen. You manage risk by being prepared and knowing your environment, but also by retaining a certain sense of humility about the best laid plans.