It’s not the best time of the week, when after long hours of preparation and intense work during a deployment window, your change fails. In addition, it also breaks other things. Sometimes it’s obvious right after deployment, but sometimes you find out hours or days later. If you’re lucky, you can apply an emergency change to fix a fairly apparent root cause, for example, one of the items missing from the deployment checklist. However, in many cases this will not be possible and you will need to revert the change.

One key to a successful withdrawal is having a plan. Yes, a backtracking plan, which is often overlooked. After all, you want your withdrawal to be an honorable surrender, not an escape from panic. To limit the damage to your business and your reputation, you must maintain control of the situation. To do that, the engineering team needs to know what to do and the Service Desk needs to keep the business informed.

A rollback plan is meant to keep you in control. It’s your insurance policy against Murphy’s Law. Let’s be honest with ourselves: we don’t insure everything. Don’t prepare a formal rollback plan for every change. Just make sure the team can verbally describe how to get back in case things go wrong.

However, you need a more formal plan for more complex changes. Creating such a plan is one of the least favorable activities of many technicians. That’s why the change manager should be responsible for doing it. It should be included with the rest of the change documentation, ready to be used if necessary.

A good backup plan should include:

• low level technical instructions,
• Specific communication instructions, with contact names.

The list of technical instructions is created by reversing the order of the activities of your implementation plan and describing how to go back in each of the executed steps. It might be relatively easy if most of the work could be accomplished by restoring the most recent backup. Consider a sample backout plan for such a scenario:

• Notify the service desk about the start of the restoration plan. (Call them, send an email or create a ticket; indicate it specifically).
• Disable user access to the system. (How? List actions.)
• Restore the backup made before the implementation of the change. (List the necessary actions).
• Perform system health checks. (List them all).
• Enable user access.
• Notify service desk of successful restitution.

Often the plan will be more complex than it seems. There may be many more restore steps, involving various databases, file systems, and other areas of the IT infrastructure. The basic template still applies. It must be detailed and adapted to each organization and each change. It goes without saying that every action must have an owner, so make sure it’s clear who does what.

Communication with the Service Desk is very important. Communication in general should be part of the plan to maintain control of the situation. In addition, the business needs to know that IT is in control. The Service Desk must take care of projecting the image of control towards the business. They can do this by issuing regular communications if the business impact is serious enough. They will also receive calls from dissatisfied users and inform them of the resolution status.

A rollback plan is your insurance policy. It’s up to you to have it or not. It is recommended to have it for every complex change, because business continuity and IT credibility are at stake. Start by preparing such a plan for the more complex change ahead. Then take advantage of that, and over time, you’ll have it ready for all the high-risk changes.