T38.1: Progressing a Tier 0 CCR

Every CCR record is associated with an Organization and a System. Understanding this hierarchy is fundamental to working with CCR:

• Organization represents the customer or company. Multiple systems can belong to a single organization. The organization that owns the system determines who appears in authorization dropdown lists (e.g., for authorizeAndStartMoveToTEST, any user from the owning organization appears as a potential authorizer).
• Responsible Organization is the entity that performs the implementation work. For InterSystems-managed systems, InterSystems is typically the Responsible Organization. For customer-managed systems, the customer organization may be the Responsible Organization.
• Site Code identifies a physical location. A single organization may have multiple sites, and each site may host multiple systems.
• System represents a specific deployed InterSystems environment. Each system has its own CCR workflow configuration, including which environments are enabled (BASE, TEST, UAT, LIVE) and which Perforce branches are associated.
• System Code (SysCode) is the short identifier used throughout CCR to reference a system. It appears in CCR record identifiers, Perforce branch names, and various reports. System codes should be meaningful and consistent.

These terms appear in CCR record creation fields and affect routing, notifications, and authorization throughout the CCR lifecycle.

---

CCR records can be created through several entry points:

1. CCR Homepage — The most common method. Click "Create CCR" from the main dashboard. 2. System Details Page — Create a CCR directly from a system's detail page, which pre-populates the System field. 3. VS Code Extension — Create CCRs directly from the IDE (more relevant for Tier 1). 4. TrakCare Integration — For TrakCare-specific workflows. 5. ISCLaunch — Automated CCR creation through scripting.

Required Creation Fields:

• System: Select the target system (determines Organization, SysCode, and available environments)
• Tier: Set to Tier 0 for non-code changes
• Title: A descriptive title for the change (e.g., "Update license key in TEST environment")
• Owner: The person responsible for implementing and progressing the CCR

Additional Creation Fields:

• Description: Detailed explanation of what the CCR covers
• Window Required for Move: Specifies the impact level when moving between environments:
• No Impact — Change can be made at any time with no disruption
• Out of Hours — Change should be made during off-hours to minimize disruption
• Downtime — Change requires system downtime; a maintenance window must be scheduled
• Related CCRs: Links to other CCRs that are related to this change

Upon creation, the CCR enters the In_PREP state in the PREP phase.

---

Proper documentation is a cornerstone of the CCR workflow. Three mandatory planning fields must be completed before progressing beyond the PREP phase:

Implementation Plan:

• Step-by-step instructions that anyone could follow to implement the change
• Should be environment-agnostic where possible, using tokens like <env> for environment-specific URLs
• Must be clear enough that a peer reviewer can verify the work was done correctly
• Example: "1. Navigate to Management Portal → System Administration → Licensing. 2. Apply new license key. 3. Verify key activation."

Backout Plan:

• Instructions to reverse the change if issues are discovered
• Critical for risk management — every change should be reversible
• Example: "1. Re-apply previous license key from backup. 2. Verify system functionality."

Testing Plan:

• Steps to verify the change was implemented correctly
• Should produce clear, documentable evidence of success
• Some testing steps may be environment-specific (e.g., certain tests should not be run in LIVE)
• Example: "1. Refresh licensing page and verify new expiration date. 2. Check Messages Log for licensing errors."

These three fields move forward with the CCR through each phase. As the CCR progresses from BASE to TEST to LIVE, the Implementation Plan, Backout Plan, and Testing Plan are displayed in the corresponding phase pane, allowing implementers to follow the same documented procedures in each environment.

Testing Steps Taken fields (one per environment: BASE, TEST, LIVE) capture the actual results of executing the Testing Plan. These fields should contain evidence such as:

• Descriptions of test results
• References to attachments (logs, screenshots, output files)
• Clear evidence of success for the peer reviewer to evaluate

---

The complete Tier 0 CCR workflow follows the BASE > TEST > LIVE progression (some systems may include UAT as well). Here is the full state progression:

PREP Phase: 1. In_PREP — Initial state after creation. Complete all planning fields (Implementation Plan, Backout Plan, Testing Plan). The CCR owner plans and documents the change.

BASE Phase: 2. In_BASE — (via markPREPComplete) Implement the change in the BASE environment. Test in BASE and document results in "Testing Steps Taken in BASE." 3. BASE_Pending_Peer_Review — (via markBASEComplete) A peer reviewer evaluates the work. The reviewer ensures documentation is complete and the change was implemented correctly. 4. BASE_Complete — (via passPeerReview) Change has passed peer review. Authorization is needed to move to TEST. 5. Pending_Manual_Move_To_TEST — (via authorizeAndStartMoveToTEST) The implementer carries out the Implementation Plan in the TEST environment.

TEST Phase: 6. In_TEST — (via markMoveToTESTComplete) Implementation in TEST is complete. Execute Testing Plan in TEST and document results. 7. TEST_Pending_Peer_Review — (via markTESTComplete) Another peer review. Reviewer verifies Testing Steps Taken in TEST. 8. TEST_Ready_For_Customer — (via passPeerReview) Customer (system owner) reviews and authorizes move to LIVE. 9. Pending_Manual_Move_To_LIVE — (via authorizeAndStartMoveToLIVE or after startMoveToLive from Pending_Move_To_LIVE) Implement the change in the LIVE environment.

LIVE Phase: 10. In_LIVE — (via markMoveToLIVEComplete) Execute Testing Plan in LIVE. Document results. Note: some testing steps may not apply in LIVE (e.g., "Do not create test patient data"). 11. LIVE_Pending_Peer_Review — (via markLIVEComplete) Final peer review. 12. LIVE_Complete — (via passPeerReview) All work is complete. Get customer authorization to close. 13. Closed — (via authorizeClosure) CCR is closed and archived. Details remain for future reference. The Clone transition can replicate the CCR for repeated changes.

Important Implementation Rules:

• In BASE: Make and test changes in the In_BASE state (iterative process)
• Beyond BASE: Only make changes in the corresponding Pending_Manual_Move_To_XXXX state
• Test the change in every In_XXXX state (In_BASE, In_TEST, In_LIVE)

---

The markPREPComplete transition moves the CCR from In_PREP to In_BASE, signaling that all planning work is done and the implementer is ready to begin making changes.

Prerequisites:

• Implementation Plan must be filled with clear, step-by-step instructions
• Backout Plan must be documented
• Testing Plan must be defined with verifiable steps
• All fields should be reviewed for accuracy and completeness

Best Practices:

• Write plans as if someone else might need to follow them — clarity is essential
• Use environment tokens (<env>, <homepage>, <smp>) so the same instructions work across environments
• Consider what evidence the peer reviewer will need to verify your work
• If the change is complex, break the Implementation Plan into numbered steps

Once in In_BASE, the owner proceeds to implement the change in the BASE development environment and test it.

---

The failPeerReview transition is used by a peer reviewer when they determine that the CCR's work is insufficient, incorrect, or inadequately documented. This is a backward transition, so Transition Notes are mandatory — the reviewer must explain what needs to be corrected.

When failPeerReview is performed:

• The CCR returns to the active implementation state for that phase
• The original owner can address the issues and re-progress the CCR
• All history is preserved in the CCR Transition History

Common reasons for failing a peer review:

• Testing Steps Taken field is empty or lacks evidence
• Implementation Plan was not followed as documented
• Documentation is unclear or incomplete
• Testing results indicate the change may not be working correctly

Important rule: The person who performed the transition into peer review (e.g., markBASEComplete) cannot perform the peer review on themselves. CCR enforces that a different user from the same organization must serve as the peer reviewer.

---

The markAcceptanceFailed transition is used at authorization checkpoints when the customer (or system owner) does not approve the change to move forward. It appears in two key states:

1. BASE_Complete — The customer rejects moving the change to TEST. The CCR returns to In_BASE. 2. TEST_Ready_For_Customer — The customer rejects moving the change to LIVE. The CCR returns to In_TEST.

After markAcceptanceFailed:

• The CCR returns to the active implementation state for the current phase
• The owner should treat this like a failed peer review: review the feedback, make corrections, and re-progress
• If significant rework is needed, the implementer may choose to:
• Fix the issue and re-progress through the workflow
• Create a catch-up CCR to address the specific concern (usually preferred)
• In rare cases, perform markValidationFailed to back out the changes entirely

Key distinction: markAcceptanceFailed does NOT back out any changes — it simply sends the CCR backward for additional work. The implementer decides whether to fix the issue in place or use a catch-up CCR.

---

The markValidationFailed transition is used when testing reveals that a change has fundamentally failed and needs to be reverted. This is different from markAcceptanceFailed — it indicates that the change itself is problematic, not just that the customer hasn't approved it.

When markValidationFailed occurs:

• The CCR moves to the Pending_Revert state
• The implementer must back out the changes according to the Backout Plan
• The CCR follows a recovery path before it can eventually be closed

Best Practices:

• Prefer catch-up CCRs over markValidationFailed — It is usually better to create a new CCR to address the issue rather than marking the original as failed. This preserves the work done and creates a cleaner audit trail.
• Use Backout Plans — This is where having a well-documented Backout Plan pays off. Follow it carefully to reverse the change.
• Document thoroughly — Record exactly what went wrong and what was done to reverse the change.
• Consider the impact — markValidationFailed affects metrics and reporting. Use it only when the change truly needs to be reverted, not for minor issues that can be addressed with a catch-up CCR.

Common scenarios for markValidationFailed:

• Change causes system instability in TEST or LIVE
• Change produces unexpected side effects that cannot be quickly resolved
• Change is incompatible with other recent changes

---

Peer reviews are a critical quality gate in the CCR workflow. They occur three times in a standard BASE > TEST > LIVE workflow:

1. BASE_Pending_Peer_Review — After implementation and testing in BASE 2. TEST_Pending_Peer_Review — After testing in TEST 3. LIVE_Pending_Peer_Review — After testing in LIVE

Why Peer Reviews Are Essential:

• Quality assurance: Ensures changes are implemented correctly and completely
• Documentation verification: Confirms that plans are accurate and testing evidence is sufficient
• Knowledge sharing: Another team member reviews the work, creating shared understanding
• Risk mitigation: Catches issues before they progress to higher environments
• Audit trail: Provides documented evidence that changes were independently verified

Peer Reviewer Guidelines:

• Verify that Testing Steps Taken contains sufficient evidence that the Testing Plan was thoroughly followed
• Ensure the Implementation Plan is accurate and was followed
• Check that all required fields are complete
• Review any attachments (logs, screenshots) for evidence of success
• If anything is insufficient, use failPeerReview with a clear explanation

Peer Review Rules:

• The person who performed the transition into peer review cannot peer review their own work
• Anyone from the same organization (other than the transitioner) can perform the peer review
• CCR automatically reassigns to the default peer reviewer when entering a peer review state
• Peer review routing can be configured at the system level (see T38.2)

---

Properly closing CCRs is the final step in the CCR lifecycle and is important for several reasons:

The authorizeClosure Transition:

• Performed from the LIVE_Complete state
• Requires specifying who authorized the closure (Close Authorized By field)
• Transition Notes can record how authorization was communicated (email, Teams, phone, in person)
• Destination: Closed

Why Closing Matters:

• Clean dashboards: Open CCRs clutter the homepage and make it harder to find active work
• Accurate reporting: Open CCRs affect metrics and aging reports
• Complete audit trail: A closed CCR represents a fully documented change from start to finish
• Future reference: Closed CCRs are the primary reference for "what changed and why"
• Cloning efficiency: Closed CCRs can be cloned to repeat similar work for the same or different systems

State: Closed:

• CCR is no longer active
• All details remain for future reference
• Edit icons remain active in case corrections are needed
• The Clone transition can create a new CCR that replicates the documentation

Best Practice: Close CCRs promptly after all LIVE testing is complete and customer authorization is obtained. Leaving CCRs in LIVE_Complete indefinitely creates unnecessary clutter and reporting inaccuracies.

---

One of the most critical rules in CCR workflow is knowing which state to perform implementation and testing activities:

BASE Environment (Special Case):

• In the In_BASE state, both implementation and testing occur together
• This is because BASE development is often an iterative process of making changes and testing
• The workflow does not require separate states for implementing and testing in BASE

TEST Environment:

• Implement the change during Pending_Manual_Move_To_TEST state
• Follow the Implementation Plan exactly as documented
• After implementation, perform the markMoveToTESTComplete transition
• Test the change during In_TEST state
• Document results in "Testing Steps Taken in TEST"

LIVE Environment:

• Implement the change during Pending_Manual_Move_To_LIVE state
• Follow the Implementation Plan carefully — note any LIVE-specific instructions
• After implementation, perform the markMoveToLIVEComplete transition
• Test the change during In_LIVE state
• Document results in "Testing Steps Taken in LIVE"
• Some testing steps may not be appropriate for LIVE (e.g., "Do not create test patient data")

Key Rule: Beyond BASE, only make changes to an environment in its corresponding Pending_Manual_Move_To_XXXX state. Never make implementation changes during In_TEST or In_LIVE — those states are for testing and documentation only.

---

CCR maintains comprehensive history tracking through two mechanisms:

Transition History:

• Records every state transition performed on the CCR
• Each entry includes: transition name, source state, destination state, user who performed it, timestamp, and any transition notes
• Provides a chronological view of the CCR's journey through the workflow
• Essential for understanding how a CCR reached its current state
• Useful for identifying who performed specific actions (e.g., who authorized the move to TEST)

Record History:

• Records all field modifications made to the CCR
• Shows what was changed, by whom, and when
• Captures edits to planning fields, descriptions, titles, and other CCR data
• Complements Transition History by showing changes made within states (not just between states)

Use Cases:

• Auditing: Verify that proper procedures were followed
• Troubleshooting: Understand why a CCR is in an unexpected state
• Compliance: Demonstrate that change control processes were followed
• Learning: Review how experienced users progress CCRs as a training reference

---

CCR supports several environment tokens that can be used in planning fields and other text areas. These tokens are automatically replaced with environment-specific values, making documentation reusable across all environments.

Available Tokens:

• <env> — Replaced with the current environment identifier. If the CCR is in the TEST phase, <env> resolves to the TEST environment's identifier.
• <homepage> — Resolves to the homepage URL configured for the current environment in System Details.
• <smp> — Resolves to the System Management Portal URL for the current environment.
• <smpPrefix> — Resolves to the SMP URL prefix (useful for constructing specific portal pages).

Usage Example: Instead of writing separate instructions for each environment:

• "Navigate to https://base-server:57772/csp/sys/UtilHome.csp"
• "Navigate to https://test-server:57772/csp/sys/UtilHome.csp"
• "Navigate to https://live-server:57772/csp/sys/UtilHome.csp"

Write once using tokens:

• "Navigate to <smp>"

Configuration: Environment-specific URLs are configured in the System Details page under Environment Details. Each environment (BASE, TEST, UAT, LIVE) can have its own:

• Namespace
• Homepage URL
• Management Portal URL
• Super Server connection details
• Terminal Port

Best Practice: Always use tokens in planning fields rather than hardcoded URLs. This ensures the same instructions work correctly in every environment and reduces the risk of errors when following the Implementation Plan in different environments.

---