Modernize Change Management

Change management is a critical component of keeping systems and applications stable in the face of continual business change and ensuring people are aware of changes being deployed. However, many organizations struggle to find the “sweet spot” between the pace of today’s business and the need to manage risk associated with fast-paced change.

The answer lies in having a strong process base combined with using today’s service management applications to their fullest capabilities. This is truly a situation in which success lies in combining people, process, and products to create an overarching process that relies on automation to ensure that approval comes from the right-sized group of people, based on potential risk.

The Basics to a Modern Approach

It’s worth laying a little groundwork before diving in. The ITIL^® Service Transition book defines the CAB in terms of its role in reviewing, authorizing, and prioritizing changes to be made and even talks about the fact that the board could be virtual.  Regardless, many organizations choose to leverage the CAB as less of a strategic body, advising on what work should/should not be performed and more tactically, just before deployment of a change to the production environment.

The approach described in this blog will not work for everyone, as several tool pre-requisites are needed before tools can be used to automate some of the required decision-making needed for this approach to succeed:

• Robust CMS/CMDB: The configuration management system or database needs to be complete, including relationships to applications and/or the business services supported by the infrastructure. It needs to be able to demonstrate conflicts, business services that might be affected by the change, and who needs to approve the deployment of changes to production from the business and IT.
• Sufficient documentation of maintenance windows and change blackout dates.
• Automated risk assessments and ability to rely on risk conditions and questionnaires to lock down the calculation of risk. Lockdown is essential as decisioning rules will be based on risk in this solution.

This approach is based on change risk and well suited to environments where DevOps is practiced: the main concept is that when you lower risk of making a change, you also lower the required approvals and due diligence needed. In this environment, all that’s really needed is the business approval to push. In fact, in a fully automated deployment environment, where approval from the business kicks off the deployment, change management should be satisfied by having the deployment tool open a change record, check for conflicts against approved scheduled changes, and start the deployment as soon as no conflicts exist. Automation can push any CI changes to the CMDB and update and close the change at the successful completion of the deployment. If a conflict is found, the change can begin as soon as the conflicting change is complete (assuming the conflict is one that would prevent successful deployment, such as network or server maintenance).

Moving beyond this will take an understanding of the current state of quality checks and planning, along with the level of communication that occurs within the organization as changes are being planned, built, and made ready for deployment as well as the amount of coordination that teams employ when performing daily work. Good communication and coordination in the culture make it easy to move towards streamlining the final approvals to deploy a change. Ideally, if teams work well together from conception until deployment, approval to deploy should be no more than a scheduling exercise.

The Program

To engage in the program of streamlining, the organization needs to review change approvals first, establishing appropriate levels of deployment authorization based on risk from very low to very high. Security review/approval may also be needed for certain types of changes and this should be able to be derived by the affected configuration items and their compliance needs or by a combination of these factors and the change category/type.

Conceptually, the next step is to build the approvals needed into the automated approval capabilities of the tool and begin to make subtle changes to the process and culture:

• Approvers need to be trained that their approval is not a rubber stamp. If they don’t see evidence of proper planning and communication, they should reject the change with notes concerning their reasons or withhold approval.
• Approvers also need to agree to respond to approval requests within timescales set by the change management team, based on how far out the change is occurring.
• The last piece is driving an understanding that changes that are unable to be approved in advance of their deployment date will need review and that lack of approval by the security team nominates a change for review automatically.

With these changes to the approval process, there are several adoption fall-outs that should be considered in planning the adoption and cultural shift needed to support the concepts on which modernization is approached:

• If the change is properly planned and coordinated from it’s inception, everything that’s needed for deployment will be ready when the business approves the change, and automated approvals are likely to enable it to go without a formal review.
• If the deployment is well planned and communicated with all members of the deployment team and properly documented in the change record, approval will be easily gained.
• Teams that fail to engage others will likely see their changes failing the automated approval process, thus requiring review or even delay.

With these steps in place, the new change review/approval process can be adopted and the need for weekly meetings significantly reduced. The new approval rule is simple:

• All changes require a risk assessment and must have no unapproved conflicts. If there are changes that are designed to happen together that register conflicts, approval of the plan to be deployed together should be placed in the change record for approver review (if needed, add an authority to approve these situations online).
• All changes that can be fully approved without formal review are authorized to proceed (this means all approvers have confidence that the change requires no discussion before deployment or that any required discussions have already occurred).
• Only very high-risk changes and changes that are not fully approved are reviewed.
• Reviews occur daily, first thing in the morning; changes are reviewed as follows:
• Very high-risk changes to be deployed within the next 24 hours
• Unapproved changes to be deployed within the next 24 hours
• Advance approval for changes within the next 48 hours
• Any very high-risk changes that are ready for review, time permitting
• Initially work towards fitting this into no more than 30 minutes per day, understanding the ultimate goal is a 15-minute meeting daily, run like a scrum call.
• As confidence is gained, remove the review requirement for very high-risk changes, enabling full approval online to signify they are well planned and good to go.
• As maturity is gained, the process should get to the point of 5-minute meetings or even days the meeting is not required.

The aspect of this blog that’s most important is killing the weekly CAB concept and returning to the concept of the CAB as a strategic board that reviews changes before work begins, ensuring alignment with business initiatives. The issue of the pre-deployment CAB is not the CAB structure, but rather the lack of agility it causes when everyone needs to wait for the meeting, and where missing deadlines causes changes to be deferred. With a daily CAB, the concept of expedited and emergency changes starts to disappear as well: the only reason for an emergency change is to restore service or for a repair that cannot wait until the next morning’s CAB cycle. Expedited changes are no longer required because changes can be approved online or daily.

So do we still need a CAB before deployment? It would be great if the answer became no, that it is eliminated as a result of great teamwork and communication. To get there, a culture of advance planning and discussion amongst deployment team members is needed. Until this happens, moving towards a short daily meeting achieves the benefits stated above: no need for expedited changes due to the ability to get a change approved online (yes, the change owner may need to talk to approvers and gain approval quickly but they no longer need a CAB meeting or expedited approval process) and, even more important, the ability to move quickly when business needs demand it.

The business benefits of this approach are tremendous, beginning with the ability to move quickly and respond to fast-changing business needs. It also starts to remove many peoples’ objections to the change process, which in many organizations lands like unnecessary bureaucracy. Additionally, it offers support benefits by requiring earlier planning and transition, along with improved communication.

Will organizations adopt this method? YES! I know of several who operate in processes that are very close to this, with the idea of taking it further. Additionally, as I propose it to clients, they understand it, and if not ready as they are deploying their new service management tool, they see it as a great step in their maturity.

Is this method controversial? YES! My last blog that asked why we need a CAB generated some great discussion via comments (many of them questioning my sanity). So, if you have thoughts, reach out to me, @msitsm on Twitter!

This has been a continuing area for heavy discussion with the DevOps and Agile community touting their methods as improved over ITIL because they are less bureaucratic. But at the end of the day, it’s not the frameworks that cause the process to become more bureaucratic, rather it’s the people managing them. The most important takeaway from this blog might be to remember why change management is performed: to support the outcomes needed by the business and to minimize risk while doing so. Automating code release along with early communication and planning go further to lower risk than the hoops many organizations make people jump through to deploy a change. 

It’s not the frameworks that cause the process to become more bureaucratic, it’s the people managing them.