T37.1: Basic Change Control Concepts

Overview

Change control is a fundamental IT discipline that governs how modifications are made to systems and environments. According to the ITIL (Information Technology Infrastructure Library) framework, a change is defined as the addition, modification, or removal of anything that could affect IT services. Change control establishes the standard procedures for managing these change requests in an agile and efficient manner, with the goal of drastically minimizing the risk and impact a change can have on business operations.

The importance of change control has grown significantly as technology, infrastructure, and software requirements continuously evolve. Solutions are becoming more complex, and InterSystems Hosted Solutions and Managed Services in specific regions now cover full-service offerings, not merely software offerings. Change control is an important component of meeting SLAs in these environments. It is part of ITIL and required for ISO 20000 certification, and it supports other processes already in use within organizations.

Distinguishing Change Control from Related Terms

At InterSystems, the term "change" appears in several distinct contexts, and it is critical to use terminology consistently for effective communication. The key terms are:

• Contract change: The process to change a contract with another group
• Customer change management: The process to change a business process for customers
• Change request: The process for an enhancement request
• Change control: The process for a change to a system or environment

This course focuses exclusively on change control in the system-and-environment sense. Confusing these terms can lead to miscommunication between teams, so consistent usage is essential.

The ITIL Foundation

ITIL stands for Information Technology Infrastructure Library and provides a set of detailed practices for IT service management. ITIL uses the term "change management" to describe what InterSystems refers to as "change control." The InterSystems courses standardize on the term "change control" and use it in place of "change management" throughout all training materials. Change control is part of ITIL and is required for ISO 20000 certification. InterSystems TrakCare Support is being certified for ISO 20000, making change control an essential component of meeting service level agreements (SLAs).

Examples of Changes

Changes can take many forms within an IT environment. Common examples include modifications to:

• Application configuration
• Disk layout
• Status of a service
• Application code
• Web server configuration
• User passwords
• User data

Each of these modifications, regardless of how minor they may appear, has the potential to affect IT services and should be governed by change control processes. A key principle emphasized in the course is that a "small change" does not equal a "small risk" -- even seemingly trivial modifications can have outsized consequences when they interact with other system components in unexpected ways.

---

Main Benefits of Change Control

Change control delivers several critical benefits to an organization. First, it evaluates the risk involved in a change before it is implemented, preventing costly mistakes. Second, it maintains records of all changes, including documented and tested implementation plans and backout plans. Third, it provides accurate and timely information about the changes to be implemented, keeping all stakeholders informed. Fourth, it establishes a formal and recorded review and approval process that creates accountability.

Additional benefits include:

• Ensuring that changes are implemented with minimum disruption to ongoing services
• Improving change prioritization so that the most important changes receive attention first
• Supporting adherence to compliance and regulatory standards
• Determining the cost and benefit associated with each change
• Improving quality and customer satisfaction

These benefits compound over time. As change control practices mature within an organization, the positive effects become increasingly visible through measurable key performance indicators.

Risks of Not Implementing Change Control

The course presents compelling real-life examples of what happens when change control is absent. In one case, a serious questionnaire performance crisis required InterSystems Support to spend several months in crisis mode tracking the root cause. Despite repeated assertions that no one had made any changes to the system, the root cause turned out to be an undocumented icon definition change that interacted with other factors to cause the crisis. Had change control been in place, this change would have been documented, and the root cause would have been identified immediately rather than after months of investigation.

In another example, a configuration change to booking restrictions caused the TrakCare location list to change drastically from one day to the next, creating a potential clinical risk by moving emergency episodes between locations. The team worked in crisis mode for a full week before discovering that a customer had made undocumented changes to booking restrictions that triggered the location list change. With proper change control, the undocumented change would have been visible immediately, reducing the crisis resolution time from a week to hours or less.

The Gut Feel Approach to Risk Analysis

The course warns strongly against the "gut feel" approach to evaluating change risk. Common rationalizations and their real-world consequences include: 1. "This change is simple, it can't fail" -- FAILURE occurs anyway 2. "I did that so many times" -- Circumstances change, leading to FAILURE 3. "It's easy, anyone can do it" -- Inexperienced implementer causes FAILURE 4. "Doesn't require any planning" -- Complications arise, leading to FAILURE 5. "It will only take 5 minutes" -- Unplanned complexity leads to FAILURE 6. "It won't impact anything else" -- Unexpected interdependency leads to FAILURE 7. "I don't need a rollback plan" -- Can't rollback quickly, leading to FAILURE

The overarching message is clear: there is no substitute for proper planning. Every change, regardless of perceived simplicity, deserves a structured evaluation through the change control process.

Benefits Demonstrated in Practice

When change control is properly implemented, the benefits are tangible and measurable. In one example, a printing crisis at a very large hospital was resolved in just 15 minutes by an overnight on-call person who was able to find and back out the offending change -- even though they had no prior knowledge of that particular change. The change control system made the change visible and reversible, turning what could have been a multi-day crisis into a brief incident.

In another example, a very large site had over 300 changes that needed to be reapplied across multiple environments as part of an upgrade process. Without change control, this manual reapplication would have taken two weeks and mistakes were likely. With change control in place, all 300 changes were reapplied from the change control application in just two hours with no issues. This demonstrates how change control transforms what would otherwise be an error-prone, time-consuming manual process into a reliable, automated operation.

---

What is Source Control?

Source control is a database for flat file items that provides central storage for code and configuration. It offers full versioning capabilities, maintaining all history and preventing permanent deletion of any version. Source control answers the fundamental questions about each change:

• Who made the change
• What was changed
• When it was changed
• Where it was changed
• Why it was changed
• How it was changed

Source control maintains every version of every item of configuration or code for a system. It allows automated merging of changes between environments and provides a snapshot of the operating environment at any point in time, enabling teams to understand and reproduce the exact state of a system at any historical moment.

Source Control as a Foundation for Change Control

While source control and change control are distinct disciplines, they are highly complementary. Source control focuses on storing and versioning the artifacts that change, while change control focuses on the process and governance around making those changes. Layering change control on top of a source control tool enables several powerful capabilities:

• Versioning of configuration items so that every state of the system is recoverable
• Easier and automated rollout of changes to target environments
• Easier and automated rollback of changes when something goes wrong
• Further automation of value-add methodologies such as build, test, and upgrade processes

This layered approach provides both the governance (change control) and the technical infrastructure (source control) needed for reliable change management.

Two Categories of Changes

The change control process should cover both types of changes:

• Versioned changes: Changes that can exist in source control, including code modifications, configuration files, and other artifacts that can be stored as files
• Documentation-only changes: Changes that cannot exist in source control, including modifications to system settings that are not file-based, manual procedures, and other changes that cannot be easily captured in a version control system

Over time, organizations should strive to move documentation-only changes to versioned changes by improving the tools used in the process. The more changes that are versioned, the more automation and reliability can be achieved. This is an ongoing maturation process -- as tooling improves, more types of changes can be brought under version control.

Source Control in Practice

At InterSystems, source control is integrated directly into the change control workflow. Code and configuration items are stored in a version control system (Perforce Helix), and the change control tools (CCR and JIRA) integrate with Perforce to link workflow records to the actual versioned artifacts. This integration enables full traceability from a change request through implementation to deployment, and allows automated rollback if a change causes problems. The InterSystems Data Platform Source Control Hooks provide the mechanism for connecting InterSystems IRIS-based development environments directly to the Perforce version control system, making source control a seamless part of the development workflow.

---

Avoiding Excessive Bureaucracy

One of the most important success factors for change control adoption is avoiding excessive bureaucracy. If the process is too cumbersome, people will find ways to circumvent it, defeating its purpose entirely. The change control system should make it easy to raise and track changes. The goal is to create a process that is thorough enough to provide protection but streamlined enough that people are willing to use it consistently. When people see change control as an enabler rather than an obstacle, adoption rates increase dramatically. The process should add value at every step rather than creating unnecessary friction.

Defining Clear Procedures

Organizations must have clear procedures defined for all types of changes. ITIL identifies three types of changes, and each requires its own documented procedure:

• Emergency changes: Changes that need to be evaluated, assessed, and either rejected or approved in a short space of time. Emergency changes should be reserved for changes intended to repair an error in an IT service that is impacting the business to a high degree or to protect the organization from a threat. Example: rebooting a crashed server.
• Standard changes: Pre-authorized changes that are low risk, relatively common, and follow a procedure or work instruction. Standard changes are not required to follow the normal change control process and can be recorded in a different way. Example: resetting a user's password. Standard changes must be low-risk, regularly occurring, well understood, and repeatable.
• Normal changes: Changes that are not emergency changes or standard changes. Normal changes follow the defined steps of the change control process. Example: fixing a bug in integration logic.

Standard changes should be managed through a catalog that defines specific criteria for when to execute each change and the work instruction for the change. This catalog should be regularly reviewed and approved. Some changes, such as data fixes, may require more frequent review. A standard change can be "demoted" to a normal change when predefined criteria are not met or when the predefined work instruction is insufficient for a particular situation.

These procedures must be communicated clearly and confirmed to be well understood by all team members involved in making changes.

Appointing a Change Control Manager

Every organization should appoint a change control manager who is responsible for overseeing change control processes within the organization. This person must be empowered to ensure compliance with change control policies. The change manager's responsibilities include:

• Reviewing and approving minor normal changes and standard changes
• Analyzing change records to identify trends
• Making sure change control processes are respected by all team members

For major or significant normal changes, a Change Advisory Board (CAB) or Emergency Change Advisory Board (eCAB) serves as an advisory committee. At a minimum, organizations should have a different implementor and reviewer for each change. This practice promotes at least two people understanding the system (having only one person understand a system is high risk) and helps catch errors through peer review, since you cannot effectively peer review your own work.

Building on Source Control and Automation

Wherever possible, organizations should build their change control processes on top of source control. This enables automated application of normal changes and automated rollback when changes cause issues. Standard changes should be automated to reduce risk further. The more automation in the change control process, the more reliable and efficient it becomes.

Key performance indicators (KPIs) that demonstrate improvement over time include:

• Increase in number of successful changes implemented
• Reduction in the number of service disruptions
• Reduction in unauthorized changes
• Decrease in average time to implement a change
• Decrease in number of disruptions (incidents, problems) caused by failed changes
• Increase in ratio of planned vs. unplanned changes
• Decrease in ratio of normal vs. standard changes (indicating more changes are being standardized)

Conversely, the top five indicators of a poor change control process are unauthorized changes, unplanned outages, a low change success rate, a high number of emergency changes, and delayed project implementations.

The "Little and Often" Principle

Organizations should adopt a "little and often" approach to change implementation. This means avoiding large batches of changes or changes that sit in development for a long period of time. Smaller, more frequent changes are easier to test, easier to review, and easier to roll back if something goes wrong. This principle aligns with modern agile and DevOps practices and reduces the risk associated with any single deployment.

Understanding and internalizing the value of change control is essential for long-term success. When all team members appreciate why change control exists and how it protects them, they are more likely to follow the processes consistently and advocate for their use within the broader organization.

---