T37.3: CCR and Primary Concepts

Overview

The CCR (Change Control Record) application is built on three foundational First Principles that guide its functionality and define the benefits of using it. These principles ensure that every change made to an InterSystems application implementation is properly managed and controlled.

The Three First Principles

Repeatable means that all changes documented in CCR can be identified and replicated in a new environment. This is critical when setting up additional environments, recovering from disasters, or migrating to new infrastructure. Repeatability also helps catch missed implications and verify that best practices are being followed. It prevents unauthorized changes from being made outside the formal process.

Revertible means that the implementation of any change must allow for that change to be undone. If a change causes unexpected problems in production, the team must be able to reverse it. CCR documentation and versioning in Perforce support revertibility by maintaining a history of all previous states.

Discoverable means that when a crisis occurs, the team can quickly identify what recent changes were made to the system. This is essential for troubleshooting production issues. When something breaks in a production environment, the first question is always "what changed recently?" CCR records provide a complete audit trail of who made what changes, when, and in which environments, enabling rapid identification of the root cause. The search and filtering capabilities in the CCR application support discoverability by allowing teams to quickly find CCRs by system, date range, owner, or state.

CCR Application Overview

CCR (Change Control Record) is a custom workflow application built on InterSystems IRIS and centrally hosted by InterSystems. It manages documentation, versioning, and flow of changes to application implementations. A "change" in the CCR context is defined as anything that could impact application functionality, whether it is a code change, a configuration change, a system setting modification, or an infrastructure update.

The term "CCR" has a dual meaning: it refers to both the application itself (the CCR application at ccr.intersystems.com) and to individual change records within it (a CCR record). CCR provides many workflow options and a robust security model where only InterSystems employees can view records and code from other organizations, ensuring data isolation between customers.

CCR can be used in any phase of a project, from early development with one environment to multi-environment workflow for business-as-usual changes. Users include anyone making, managing, or searching for changes made to an implementation, such as developers, project managers, and support personnel. Organizations that use CCR include InterSystems, implementation partners and third-party consultants, and end-user customers.

CCR Functionality

The key functional capabilities of CCR include allowing thorough documentation with fields for users to complete and automatically generated audit trails, interacting with Perforce on behalf of the user, facilitating movement of changes between Perforce and environments, and providing formal and recorded peer review and authorization to move changes.

Why Every Change Matters

A key CCR principle is that "small change does not equal small risk." All changes should go through the CCR process, including those involving non-exportable items. Even seemingly minor changes can have significant impacts on application functionality. A one-line configuration change, a single table modification, or a minor system setting adjustment can all cause outages or data corruption if not properly documented and controlled. The CCR process ensures that every change, regardless of perceived size or risk, is documented, reviewed, and trackable.

---

Overview

CCR defines four possible Primary Environments that represent stages in the change management workflow. Not all environments are required for every system, but the workflow order between them is fixed and must be followed.

The Four Primary Environments

BASE is the initial environment where all changes originate. Developers and implementers make their changes in the BASE environment first. The only exception is LIVE-only Systems, where changes are made directly in production. When a system has two or more environments, one must always be designated as BASE.

TEST is the testing environment used for validation. After changes are completed and documented in BASE, they are moved to TEST to verify both the process of moving changes (which can itself fail) and the functionality of the changed items. Isolating changed items in TEST helps verify they meet requirements and specifications.

UAT (User Acceptance Testing) is a secondary testing environment. It provides an additional validation layer where end users or business stakeholders can verify that changes meet their expectations before the changes reach production.

LIVE is the production environment. This is the final destination for all changes and represents the system that end users interact with daily. Non-BASE environments in multi-environment workflows are locked from manual changes outside the CCR process and require customer authorization to receive changes.

Change Flow Through Environments

Changes always flow in a fixed order through the configured environments: BASE to TEST to UAT to LIVE. This order cannot be skipped or reversed. The sequential flow ensures that every change is validated at each stage before reaching the next one. For example, a change that fails in TEST will not proceed to UAT or LIVE, protecting the production environment from untested modifications. The workflow diagram shows the environments connected by arrows indicating the direction of change flow.

Environment Configurations

A CCR System can be configured with 1, 2, 3, or 4 Primary Environments. With a single environment, it must be either BASE or LIVE only. With two or more environments, one must be BASE. The most common configuration is three environments: BASE, TEST, and LIVE. A four-environment configuration adds UAT between TEST and LIVE, providing an additional validation layer. The number and configuration of environments is defined in the CCR System settings and determines the workflow that all CCRs in that system must follow.

Why Multiple Environments

Multiple environments serve two critical purposes. First, the process of moving changes between environments is itself tested, since moving a change can fail. Second, the functionality of the changed items is tested in isolation, since a change can fail to meet requirements or specifications.

CCR System Definition

A CCR System encompasses all environments included in the workflow for a particular application. Systems can be defined to have 1, 2, 3, or 4 Primary Environments. A CCR Environment is the hardware and software that provides a standalone copy of functionality for an application. For example, one namespace of IRIS for Health with application code and configuration on a server running Red Hat constitutes an environment.

Environment Rules and Constraints

When using a single environment, the system must be configured as either BASE only or LIVE only. With two or more environments, one must always be designated as BASE. Non-BASE environments in multi-environment workflows are subject to two important constraints. First, they are locked from manual changes outside the CCR process, meaning no one should be making direct changes to the TEST or LIVE environment without going through the formal workflow. Second, they require customer authorization to receive changes, providing an additional layer of approval before changes reach testing or production environments.

---

Overview

CCR Transport is the mechanism that moves changes between environments, the CCR application, and Perforce. It is the automated process that ensures changes flow correctly through the defined workflow of environments.

How CCR Transport Works

CCR Transport is tightly integrated with Perforce and automates the complex process of moving changes. Without CCR Transport, developers would need to manually export changes, submit them to Perforce, and then manually import them into each subsequent environment. This manual approach is error-prone and does not scale. CCR Transport handles all of this automatically, ensuring consistency and reliability.

The transport process follows a specific sequence:

1. Updates are made in the BASE environment by the developer or implementer 2. The updates are flagged as changed within the CCR application 3. The changed items are exported from the BASE environment 4. The exported items are transported from BASE into Perforce (the source control repository) 5. Items are merged through Perforce using automated processes managed by CCR 6. Items are transported out of source control and into the next environment in the workflow (for example, TEST) 7. The items are imported and deployed in the target environment

This process repeats for each subsequent environment. After testing and validation in TEST, the items can be transported to UAT (if configured), and finally to LIVE. At each step, CCR manages the interaction with Perforce on behalf of the user, so users do not need direct access to Perforce tools.

The Role of Perforce

Perforce Helix is the commercial Version Control System (VCS) used by CCR. It serves as a database for flat file items and provides central storage for code and configuration. Perforce provides full versioning capabilities, meaning every version of every file is retained and can be retrieved.

CCR uses Perforce to version exportable changes (code and configuration), automate the merging of changes through the workflow, and maintain a complete copy of the code in each environment. Customers using CCR receive Perforce licenses, but they can only access Perforce through the CCR application. Direct access to Perforce tools is not provided to customer organizations; all interactions with Perforce are mediated by the CCR application, which ensures proper workflow enforcement and audit trail maintenance.

Transport and CCR Tiers

CCR Transport is not used at Tier 0 (documentation only). At Tier 0, changes are implemented manually in all environments -- the developer must replicate their changes by hand in TEST, UAT, and LIVE. Transport is introduced at Tier 1 and Tier 2, where source-controlled items need to be moved automatically between environments.

Transport Failure Handling

One of the reasons for having multiple environments is that the transport process itself can fail. A change that works correctly in BASE may encounter merge conflicts, missing dependencies, or configuration differences when transported to TEST. By testing the transport process in TEST before LIVE, issues can be identified and resolved without impacting production users.

CCR Transport and the Principle of Little and Often

The principle of "little and often" is critical to successful usage of CCR and directly impacts how Transport works. Users should minimize the number of changes in a single CCR because all items in a CCR progress through the workflow together, making it difficult to isolate one change and progress it separately. Keeping CCRs small reduces the probability of blocked workflow due to dependencies, enables clear documentation of the impact of a CCR, and results in better titles and descriptions due to more narrowly defined changes. Users should also progress CCRs quickly but safely -- do not allow CCRs to remain mid-workflow for too long (weeks), and minimize the time between starting a change and completing it in all environments.

---

Overview

Understanding Perforce terminology is essential for working with CCR at Tier 1 and above. Three key terms define how changes are organized and tracked in the Perforce version control system that underpins CCR.

Perforce Branch

A Perforce Branch in the CCR context is the depot path associated with a particular CCR System. It represents the location in the Perforce repository where all source-controlled files for that system are stored. For example, a system with the code "LSApps" might have its Perforce Branch set to //custom_ccrs/us/ISCX/LSApps/. The branch contains the complete codebase for the system and each environment maps to a location within or derived from this branch. The Perforce Branch is visible on the CCR System Details page.

Item

An Item is a single file or artifact that is tracked in Perforce as part of a CCR change record. Items are the individual units of change. Examples of items include ObjectScript classes, routines, CSP pages, JavaScript files, DTL transformations, integration logic, and production configurations. When a developer modifies a file in the BASE environment and associates it with a CCR, that file becomes an Item on the CCR record. Each modification to an Item creates a new version in Perforce, providing a complete change history.

ItemSet

An ItemSet is a collection of related Items that are grouped together and transported as a unit within a CCR. When a CCR record contains multiple changed files, those files form an ItemSet. The ItemSet represents the complete set of changes that must be moved together through the workflow. All items in an ItemSet progress through the workflow together, meaning all or none of the changes advance to the next environment. The System Details page includes an "Undeployed Itemsets" tab that shows ItemSets waiting to be deployed.

Relationship Between These Terms

A CCR System has a Perforce Branch. When changes are made in the BASE environment and associated with a CCR record, individual changed files become Items. Multiple Items on a single CCR form an ItemSet. When the CCR is authorized to move to the next environment, the entire ItemSet is transported through Perforce from one environment branch to the next.

It is important to note that all Items in an ItemSet progress through the workflow together. This means that if one Item in a set has a problem, the entire ItemSet is held back. This reinforces the "little and often" principle: keeping CCRs small with fewer Items per ItemSet reduces the risk of blocking the entire set.

The Recent Changelists tab on the System Details page shows the Perforce changelist history, providing visibility into what has been transported and when.

---

Overview

CCR provides different tiers of usage that offer increasing levels of automation and control. The tiered approach provides flexibility for many types of applications and allows projects to adopt CCR at the level appropriate for their needs.

Tier 0 - Documentation and Workflow Tool

Tier 0 is the foundational level of CCR usage. At this tier, CCR is used purely for documentation and workflow management. No record of the change is maintained in source control (Perforce). Changes are implemented manually in all environments, and if a change needs to be backed out, that is also a completely manual task.

Example Tier 0 changes include file system path changes, memory settings, user creation, patching, upgrades, and any manual change to the environment or system. Standard changes should all be CCR Tier 0 changes, as standard changes should never involve custom code stored in source control.

A solid understanding of Tier 0 usage and best practices is a prerequisite for using CCR at higher tiers. Except for the extra steps related to CCR Transport, the workflow of Tier 0 CCRs is identical to Tier 1 and Tier 2 CCRs.

Tier 1 - Source Control Automation and Code Deployment

Tier 1 adds source control automation and automated code deployment. At this tier, source code is stored in Perforce, and CCR Transport automates the movement of code between environments. This eliminates the need for manual code deployment and ensures that the exact same code is deployed to each environment in sequence.

Example Tier 1 changes include classes, routines, reports, CSP pages, JavaScript files, integration logic, DTL transformations, and productions. These are all items that can be exported as flat files and stored in the Perforce version control system. The automation provided by Tier 1 significantly reduces deployment errors and provides a complete versioned history of all code changes.

Tier 2 - TrakCare Configuration Transport and Deployment

Tier 2 extends the capabilities of Tier 1 to include TrakCare-specific configuration items. At this tier, configuration data is stored in Perforce and can be automatically transported and deployed between environments. This is particularly valuable for TrakCare implementations where configuration changes are frequent and complex.

Example Tier 2 changes include TrakCare code tables, layouts, security groups, code table changes, and workflow configuration. This tier is specific to TrakCare implementations and provides the highest level of change control automation. By storing configuration in Perforce alongside code, Tier 2 enables a unified change management process for all aspects of a TrakCare deployment.

Progressive Nature of Tiers

The tiers are progressive in terms of both complexity and benefit. Each higher tier builds on the previous one and includes all capabilities of the lower tiers. A Tier 2 system, for example, uses all of the documentation and workflow capabilities of Tier 0, all of the source control automation of Tier 1, and adds TrakCare-specific configuration transport on top. Projects should determine their appropriate tier of usage based on the types of changes they need to manage and the level of automation they require.

Documentation and training applicable to a tier should be fully understood prior to enabling that tier. The ICC (InterSystems Change Control) training courses are organized by tier, with the ICC300 series covering Tier 0, and separate course series covering Tier 1 and Tier 2 topics. The CCR workflow is best first learned on Tier 0 records, which are simpler to understand because they do not require CCR Transport. Changes are versioned and stored in Perforce at Tiers 1 and 2, but not at Tier 0.

Tier 0 as Foundation for All Tiers

A thorough understanding of CCR Tier 0 principles is a requirement for all CCR users, regardless of the tier they will primarily work at. The courses focused on CCR Tier 0 are a foundation for everything else related to proper CCR usage. Tier 0 material covers the default workflow, best practices around data collection, non-default workflow paths, project management features (merging, grouping, PM reports), initial system definition, and peer review best practices. Except for the extra steps related to CCR Transport, the workflow of Tier 0 CCRs is identical to Tier 1 and Tier 2 CCRs.

---