T39.3: Best Practices and Debugging Techniques

Why Catch-up CCRs Are Needed

Once a CCR has progressed past the BASE_Complete state, it is no longer possible to bundle and upload additional changes to that CCR. This restriction exists because any changes made after BASE_Complete have not been peer reviewed in the BASE environment. If additional modifications are discovered as necessary (for example, a bug fix or a missed item), you must use one of two approaches to get those changes into the workflow.

Option 1: Catch-up CCR (Recommended)

A catch-up CCR is a new CCR created specifically to carry the additional changes that were missed or need to be added after the original CCR progressed past BASE. This approach is strongly encouraged because it simplifies change record documentation and avoids the complexity and risk of backing out changes.

Steps to create and use a catch-up CCR: 1. Create a new CCR or clone the original CCR and trim details as required 2. Specify the original CCR as a Prerequisite CCR 3. Use the Title field to clarify it is a catch-up CCR (e.g., "[Catch up to ISCX12345]: Fix typo in username label") 4. Progress the catch-up CCR to In_BASE state 5. Make and upload the additional changes (or upload changed items that were left behind) 6. Document, test, and progress the catch-up CCR to the same state as the original CCR 7. Merge the catch-up CCR into the original CCR (source is catch-up, target is original) 8. Progress the original CCR -- this will integrate all Perforce changes from both the original and catch-up CCR

Option 2: Backwards Transitions (Not Recommended)

The alternative is to use backwards transitions to return the CCR to the BASE state. This approach requires backout if the CCR is past the BASE phase, is usually more work than creating a catch-up CCR, and can cause merge conflicts. Resolving merge conflicts is time consuming and can introduce risk. Additionally, it makes it more difficult to understand whether a change has ever made it to TEST or UAT environments, since you must use transition history instead of just noting the state of the CCR. Always preview backout and verify no merge conflicts before performing any backwards transition.

---

What Revision History Shows

For any file in the Perforce repository, the revision history provides a complete chronological record of every change made to that item. For each revision, it displays:

• The revision number of the item
• The CCR ID associated with that changelist
• Whether the associated CCR is currently active (active means not in Closed, Merged, or Cancelled state)
• Other CCR details relevant to understanding the change context

How to Access Revision History

1. Open the CCR of interest 2. Click the Submitted Changes tab in Perforce Details 3. Click the revision history icon (clock icon) next to the item of interest 4. The revision history window displays the complete history of that file across all CCRs

When to Use Revision History

Revision history is particularly valuable for:

• Identifying collision risks: Spotting other active CCRs that are modifying the same item as the current CCR, which could lead to merge conflicts
• Understanding change frequency: Determining how often an item is being modified, which may indicate a hotspot file that requires extra coordination
• Spotting anomalies: Identifying irregularities such as a changelist without a Perforce job, which could break future integration operations
• Investigating merge conflicts: When a merge conflict occurs, revision history helps identify which other CCR made the conflicting change

---

Where to Find Error Messages

Error messages and alerts in CCR Transport appear in the Perforce Details section of a CCR. The Perforce Alert field displays the current alert status, which may be a warning or an error. These alerts are the primary mechanism for communicating transport problems to users.

Types of Alerts

• WARNING: Typically generated by a preview integration. For example: "WARNING: Conflicts predicted. One of these blocked integration to LIVE: ISCX10582, ISCX10589." This indicates potential conflicts that should be resolved before progressing.
• ERROR: Generated when an actual transition fails. For example: "ERROR: Critical error occurred. Integration Blocked by: ISCX9321. Aborting ItemSet creation." This indicates the transition was blocked and the CCR could not progress.

Interpreting Alert Details

• When a single CCR ID is specified in the error message, that CCR is definitively the source of the problem
• When two or more CCR IDs are specified, at least one of them is the source of the problem, but you must investigate further to determine which one
• Non-preview integrations that fail are shelved for InterSystems support (shelving means temporarily stored but not committed; shelved changelists can be safely ignored)

Clearing Alerts

Alerts in Perforce Details are usually cleared automatically through corrective action, such as a successful integration. For alerts that are not cleared automatically, you must clear them manually after the corrective action is completed: 1. Open the transport log 2. Click the [clear alert] link 3. Cleared alerts remain visible in the transport log for audit purposes and inform other users that the alert is no longer an issue

---

What Transport Logs Contain

Transport logs provide a comprehensive record of all Perforce-related actions that have occurred against a CCR or its ItemSets. The log captures:

• ItemSet uploads/commits: Records of when ItemSets were uploaded and committed to the Perforce branch
• Perforce integrations: Details of integration operations between branches (e.g., BASE to TEST, TEST to LIVE)
• ItemSet creation for download: Records of when ItemSets were created for deployment to environments
• ItemSet load log: Records uploaded from the client showing what happened when an ItemSet was loaded

How to Access Transport Logs

The transport log is accessed from the Perforce Details section of a CCR: 1. Navigate to the CCR's Perforce Details section 2. Click View to display the log in the browser, or Download to save it as a file 3. When viewing in the browser, click the jump to bottom button to access the most recent entries, as the log is chronological with the newest entries at the bottom

Using Transport Logs for Error Investigation

When a Perforce integration error occurs, the transport log contains critical diagnostic information:

• The log shows the specific files involved in the conflict
• It indicates the number of conflicting diff chunks for Perforce integration errors (e.g., "Diff chunks: 8 yours + 1 theirs + 0 both + 1 conflicting")
• It identifies possible sources of the conflict by listing changelists and their associated CCR IDs
• Error messages such as "ERROR #5001: ERROR: cannot resolve conflicts, unable to perform automatic integration" provide clear indication of what failed

Reviewing the transport log is also an excellent way to better understand how CCR automation works, as it traces every automated action the system performs during the lifecycle of a CCR.

---

What Causes Merge Conflicts

Merge conflicts occur when two or more CCRs modify overlapping areas of the same file. The typical scenario is: 1. CCR A makes a change to a file in BASE 2. CCR B makes a change to the same area of the same file in BASE 3. When one CCR attempts to integrate (progress) to a higher environment like TEST, Perforce cannot automatically reconcile the conflicting changes

Detecting Conflicts with Preview Integration

Preview integration allows you to check for potential conflicts without actually performing the integration: 1. Navigate to Perforce Details > Perforce Integration tab 2. Verify the menu for integration environments is correct 3. Select the Preview checkbox 4. Click Integrate 5. The preview will not commit the integration but will return any predicted conflicts

Resolving Conflicts Discovered on Preview

When a merge conflict is discovered through preview integration: 1. Edit CCR A to specify CCR B as a prerequisite 2. Progress CCR B ahead of CCR A in terms of phase 3. Once CCR B has been integrated, the conflict will be resolved and CCR A can proceed

Resolving Conflicts on authorizeAndStartMoveToXXXX

When a merge conflict occurs during an actual transition (authorizeAndStartMoveToXXXX), CCR B is specified as blocking CCR A: 1. Perform the markIntegrationFailed transition on CCR A 2. Edit CCR A to specify CCR B as a prerequisite 3. Progress CCR B to the next environment 4. Optionally perform preview integration on CCR A to confirm all merge conflicts are resolved 5. Progress CCR A (CCR B must progress ahead of CCR A in terms of phase)

Circular Dependencies

A circular dependency occurs when two or more CCRs are dependent on each other. This happens when both CCRs are in In_BASE state and make overlapping changes to each other's files. The solution is typically to merge the CCRs. If merging is not possible, manual intervention by InterSystems support might be required.

Preventing Merge Conflicts

Do not use class descriptions for change documentation, as class descriptions with embedded change logs frequently cause merge conflicts when multiple CCRs modify the same class. Keep all change documentation within the CCR itself.

---

What Is Misalignment

Misalignment occurs when a CCR that is editing an existing item attempts to progress ahead of the CCR that originally added that item. This is a specific type of prerequisite conflict that the CCR system detects and prevents.

Why It Cannot Be Allowed

When an edit action is integrated before the corresponding add action, Perforce converts the edit into a branch action (which is similar to an add). This has a critical consequence: if a backout becomes necessary, the branch action would be backed out as a deletion, causing the unintentional removal of the item from the target environment. The system prevents this to ensure that only the CCR that originally added the item has the ability to delete it during a backout.

Detection and Error Messages

• On preview integration: The system generates a "WARNING: Prerequisite CCR detected" message. The solution is to stop and not perform authorizeAndStartMoveToXXXX until the adding CCR progresses with an add action to the next environment.
• On authorizeAndStartMoveToXXXX: The system generates "Critical error occurred. Another CCR has been detected as a prerequisite and marked as such. Aborting ItemSet creation." The solution is to follow the standard steps for merge conflict resolution -- perform markIntegrationFailed, set the adding CCR as prerequisite, progress the adding CCR first, then progress the editing CCR.

---

Backout Overview

When changes need to be removed from one or more environments, the backout process: 1. Backs out changelists from the highest Perforce branch to which the change has progressed 2. Integrates the backout to other affected Perforce branches 3. Creates ItemSets based on the Perforce branches for each affected environment 4. These ItemSets must be deployed to the corresponding environments like any other ItemSet

Automatic Backout

CCR automatically backs out all changes from all environments for Tier 1 CCRs only during the following transitions:

• cancel
• revert
• changeSpec

ItemSets for each affected environment are generated automatically if the Perforce backout is successful. All generated ItemSets must be deployed, and the CCR should be progressed through the next transition after successful ItemSet deployment.

Manual Backout Without Automatic Backout

For situations where automatic backout does not apply (Tier 2 CCRs, markValidationFailed transitions on Tier 1 CCRs, or other scenarios): 1. Navigate to Perforce Details > Perforce Backout tab 2. Choose the furthest environment to which changes have progressed (the default is usually appropriate) 3. The system automatically integrates the backout to all prior environments and creates ItemSets for all affected environments 4. Use Preview Only to check for potential merge conflicts before executing the actual backout 5. Deploy all generated ItemSets to their respective environments

For example, choosing TEST will backout from TEST and create ItemSets for both TEST and BASE.

Restore to BASE Flag

The Restore to BASE flag on the manual backout controls how BASE environment changes are handled:

• Cleared: Removes changes related to this CCR from all environments including BASE. Use this for cancel or changeSpec transitions.
• Selected: Maintains changes related to this CCR in BASE. The system integrates the backout changelist to BASE then backs it out to reintroduce the change. Use this for markValidationFailed transitions. This still generates an ItemSet that must be deployed (version information of items is updated).

Backout Errors

If backout errors occur, contact InterSystems support. They may need to intervene using p4v (the Perforce visual client) to resolve the issue directly at the Perforce level.

---

Understanding Bundle and Upload Validation

When an ItemSet is bundled and uploaded to a CCR, the system performs validation checks before committing the changes to Perforce. These checks ensure that the items are properly formatted, correctly associated with the CCR, and compatible with the target Perforce branch. If validation fails, the ItemSet is not committed and an error is generated.

Common Validation Issues

Validation errors can arise from several sources:

• Items that do not match the expected format or structure for the target branch
• Missing dependencies where an item references another item that does not exist in the branch
• Attempts to upload items that conflict with the current state of the Perforce branch
• Client Tools version mismatches that cause incompatibilities during the upload process

Resolving Validation Errors

To resolve bundle and upload validation errors: 1. Review the error message in the Perforce Alert and the transport log for specific details about what failed 2. Identify the root cause of the validation failure 3. Fix the issue in the source environment (correct the item, add missing dependencies, etc.) 4. Re-bundle the corrected items into a new ItemSet 5. Re-upload the ItemSet to the CCR

Maintaining CCR Client Tools

Always update CCR Client Tools through the same System to maintain accurate version history in Perforce. Best practice is to configure a separate System for the %SYS namespace for changes that affect the entire instance (such as hardware or memory allocations) and for maintaining CCR Client Tools. Alternatively, choose one existing System for such updates (for example, HSCUSTOM on HealthShare systems). To update Client Tools, go to the appropriate System Details page, click Update Client Tools (only Perforce users can do this), which creates a Tier 1 CCR that integrates current client tools to the BASE branch. Progress the generated CCR through the normal workflow and deploy the generated ItemSet to BASE.

---

How Changelists Are Associated with CCRs

In CCR Transport, Perforce changelists are associated with CCRs through the Perforce Job field. When an ItemSet is uploaded, the job for each changelist is set to the ID of the CCR. This job association is what links the code changes in Perforce to the corresponding change record in the CCR application.

When Changes Go to the Wrong CCR

There are several scenarios where a changelist may need to be moved from one CCR to another:

• An ItemSet was uploaded against the wrong CCR by accident (the user specified the wrong CCR ID on the Bundle and Upload screen)
• A CCR was cancelled by mistake and the changelists were not backed out. The user wants to attach the changelist to a new CCR to progress the change.

Resolution Options

The CCR application itself does not have built-in functionality to change the job associated with a changelist. Moving a changelist to a different CCR requires direct Perforce intervention:

For InterSystems employees:

• Use p4v (Perforce Visual Client) or Swarm (Perforce web interface) to change the job field of the affected changelist
• This requires access to the internal InterSystems network

For customers:

• Customers do not have direct access to p4v or Swarm on the InterSystems Perforce server
• Customers must contact InterSystems support to request that the changelist be reassociated with the correct CCR
• Provide the changelist number, the incorrect CCR ID, and the correct CCR ID to expedite the request

Important Considerations

• Moving changelists between CCRs is an administrative action that bypasses normal CCR workflow controls
• Ensure the target CCR is in the appropriate state (typically In_BASE) to receive the reassociated changelist
• After the job is changed, verify the changelist appears under the correct CCR's Submitted Changes tab
• The transport log of the original CCR will still contain records of the original upload

---