We’re all familiar with the risk-averse adage, “if it ain’t broke, don’t fix it,” and in engineering, it IS often wise to avoid tinkering with things when there is no good reason to risk the unexpected. Unfortunately, this is a stifling environment for creative professionals that misses out on genuine opportunities to improve service levels. I’ve found it handy to have a few things “ready to go” in a crisis…actions that fall into two categories:

First, there are the “we’ll do that the next time we have to reboot the server,” tasks. It’s helpful to have a set of tasks like this around for when there is downtime–while you are waiting for the phone call from vendor support or while the entire office is offline due to something beyond your control such as a localized communications outage. “Since no one can access the internet for the next couple hours…we will be performing some systems maintenance tasks…” These kinds of tasks should be well-contained and unrelated to the crisis at hand–i.e. don’t start things you can’t finish.

But the big opportunity presented by a crisis is the chance to try a new business process–after the immediate crisis is covered. The “Why?” is fresh in people’s minds. There will be a receptive audience asking, “how can we prevent a problem like this from happening again?” The IT department–usually not highly visible to the rest of the business–now has the attention of everyone, for good or bad. Have something good to propose and be ready to use the crisis to support your arguments for change.

In one company for whom I worked, a series of service disruptions provided the motivation to finally achieve management support for a platform upgrade. We had been running many websites on many servers with divergent code bases and rotating pagers through three people who had become accustomed to waking every night at 3am to restart IIS or maybe even reboot a server. We tried many things to work within the confines of the old systems–and we communicated what we were doing all along so management knew we were doing everything we could. But finally, a major outage in the middle of the day–as the company was trying to find customers and investors to avoid going out of business–provided the impetus to agree on a total development freeze to allow 3 months for platform migration.

It is a delicate balance. Nobody wants to hear an IT department complaining that they can’t do anything and that they need a crisis to act. Or worse, to present your ideas for solving problems and have a reaction like, “Well that’s great. Why haven’t you already done this?? Oh, sorry, we still don’t have any money.” So it is important, when you play the “now is the time to act!” card to have established a credible basis of competence…the systems ARE being held together with duct tape and aggressively monitored with the best application and performance monitoring systems you can find–and you can keep doing that…but even if you optimize your response time so that you can fix things in 10 minutes every time, users are going to notice and service levels will be affected.

When you encounter a situation that has you thinking, “I can’t believe they do this!” get over the frustration and plan for the day when the barriers to change can be blown away by crisis. Tinker on the development server and have a solution well-baked so you can roll it out: “to address performance and reliability problems, we are launching a new system…please be patient while we work out any new problems.” Crisis is not the time for experimentation: “We’ll try upgrading the software and see if that fixes it.” But it can be a time to step up and lead an organization out of the wilderness…to recognize…it really was broke, and now we need to fix it.