T9.4: Upgrades Existing Installation

Overview

Upgrading a UCR federation is a multi-step process that requires careful planning across all components. Unlike a single-system upgrade, a UCR upgrade must coordinate across the Registry, all Edge Gateways, Access Gateways, Bus, and ODS instances. Thorough planning minimizes downtime and reduces the risk of issues during the upgrade.

Pre-Upgrade Checklist

InterSystems provides a comprehensive pre-upgrade checklist that includes:

• Verifying current system health and resolving any existing issues
• Reviewing release notes for the target version
• Identifying all custom code, CDA transforms, and configuration changes
• Documenting current configuration settings for each component
• Ensuring adequate disk space on all instances for the upgrade process
• Verifying backup procedures and testing restore capabilities
• Obtaining new license keys if required by the target version
• Coordinating with facility stakeholders on maintenance windows

Release Notes Review

Release notes should be reviewed for:

• Breaking changes that may affect existing configurations or customizations
• Deprecated features that need to be replaced
• New features that may benefit the deployment
• Known issues and workarounds
• Required pre-upgrade and post-upgrade steps specific to the version
• Database schema changes that may affect custom queries

Upgrade Tools and Resources

InterSystems provides:

• Upgrade documentation specific to each version transition
• Pre-upgrade validation utilities that check system readiness
• Migration tools for configuration and data conversion
• Support channels for upgrade assistance
• Test environments or sandboxes for upgrade rehearsal

Upgrade Order

Components should be upgraded in a specific order: 1. Registry (Hub) first 2. Edge Gateways (including Audit Edge) 3. Access Gateways 4. Bus 5. ODS

---

Overview

Before upgrading an Edge Gateway, it must be properly prepared to ensure no data is lost and the upgrade proceeds smoothly. This preparation involves stopping incoming data flows, allowing current processing to complete, and creating backups.

Turning Off Incoming Connections

The first step is to stop new data from entering the Edge Gateway:

• Disable or pause business services that receive data from source systems
• Notify source system administrators that the data feed will be temporarily interrupted
• If using HL7 or other message-based feeds, stop the listener services
• Verify that no new messages are being received

Draining In-Flight Messages

After stopping incoming connections:

• Monitor message queues for all business hosts
• Allow currently queued messages to complete processing
• Verify that all queues are empty or at acceptable levels
• Check for any messages in error status that need resolution before upgrade

Stopping the Production

Once queues are drained:

• Stop the Edge Gateway production gracefully through the Management Portal
• Verify the production has fully stopped (all business hosts show stopped status)
• Document the final state of all business hosts for reference during post-upgrade restart

Creating Backups

Before proceeding with the upgrade:

• Create a full backup of all databases in the Edge Gateway namespace
• Back up the iris.cpf configuration file
• Back up any custom code, transforms, or configuration files
• Verify backup integrity and store backups in a secure location
• Document the backup location and contents

---

Overview

License keys are often version-specific, meaning that upgrading to a new version of IRIS for Health or UCR may require new license keys. All required keys must be obtained before the upgrade begins to avoid delays during the process.

Gathering Process

• Contact InterSystems or use the license portal to request keys for the target version
• Provide server identification information for each instance
• Specify the component type and required capacity for each instance
• Request keys for all instances: Registry, all Edge Gateways, all Access Gateways, Bus, and ODS

Key Validation

Before starting the upgrade:

• Verify each license key file is valid and not corrupted
• Confirm the key matches the server hardware or virtual machine it will be applied to
• Check that the key enables all required features for the component type
• Verify the key expiration date extends beyond the planned upgrade date
• Test keys on a non-production instance if possible

Key Management During Upgrade

• Keep all license key files accessible on the servers where they will be applied
• Have backup copies available in case of file corruption
• Document which key corresponds to which instance
• Plan for the key application step in the upgrade procedure

---

Overview

The core of the UCR upgrade is running the InterSystems installer on each IRIS for Health instance. The installer performs an in-place upgrade, replacing system binaries and updating system databases while preserving user data and configuration.

Upgrade Execution Steps

For each instance in the federation (in the planned order): 1. Stop the instance if not already stopped 2. Run the new version InterSystems installer 3. Select the upgrade option (the installer detects the existing instance) 4. Apply the new license key when prompted (or after installation) 5. Allow the installer to complete system database upgrades 6. Verify the instance starts with the new version

What Gets Upgraded

• System executables and binaries
• System databases (IRISSYS, IRISLIB, IRISAUDIT structure)
• System classes and libraries
• Management Portal interface
• Web gateway components

What Gets Preserved

• User databases and their contents
• The iris.cpf configuration file (with possible automatic updates for new settings)
• Custom code in user namespaces
• SSL/TLS configurations
• Production configurations (business hosts, settings, schedules)

Upgrade Considerations

• Ensure adequate disk space for the upgrade (the installer may need temporary space)
• Monitor the upgrade log for errors or warnings
• Do not interrupt the upgrade process once it has started
• If the upgrade fails, restore from backup rather than attempting to re-run
• Keep the previous version installer available in case rollback is needed

---

Overview

After the software upgrade completes on an instance, the UCR namespaces must be reactivated. This process verifies that configurations are intact, databases are properly mapped, and the namespace is ready to resume operation.

SSL/TLS Verification

After upgrade, verify SSL/TLS configurations:

• Check that all SSL/TLS configuration entries still exist in the Management Portal
• Verify certificate files are still accessible and valid
• Test SSL/TLS connections to other components (if they are already upgraded and running)
• Update any SSL/TLS settings if the new version introduces changes to security defaults

Parallel Fetch Confirmation

Parallel fetch settings control how the Access Gateway queries multiple Edge Gateways simultaneously:

• Verify parallel fetch is still enabled after upgrade
• Confirm the parallel fetch settings (timeout, maximum parallel queries) are correctly configured
• Test parallel fetch behavior with a sample query if Edge Gateways are available

Namespace Reactivation Steps

The reactivation process includes: 1. Open the Management Portal and navigate to the UCR namespace 2. Verify database mappings are correct (the upgrade may have modified system mappings) 3. Run any namespace reactivation utilities provided for the target version 4. Verify that all required globals and routines are accessible 5. Check that production configuration is intact and business hosts are properly defined

Post-Reactivation Verification

After reactivation:

• Review the namespace for any error indicators
• Verify that custom globals and data tables are accessible
• Check that package mappings are correct
• Confirm that web applications associated with the namespace are functional

---

Overview

Custom code developed for the UCR deployment must be recompiled after a version upgrade because the underlying system classes and libraries have changed. Additionally, any customizations to CDA XSLT transforms must be carefully migrated to the new version's base transforms.

Custom Code Recompilation

After upgrade, custom code must be recompiled:

• Navigate to each namespace containing custom classes
• Compile all custom classes using the Management Portal (System Explorer > Classes > Compile)
• Alternatively, use the command line: `do $system.OBJ.CompileAll()`
• Monitor compilation for errors indicating API changes or deprecated methods
• Resolve any compilation errors before proceeding

CDA XSLT Migration

CDA (Clinical Document Architecture) transforms may have been customized to modify document rendering:

• Compare custom XSLT modifications with the new version's base XSLT files
• Identify changes in the base XSLT that may conflict with customizations
• Merge custom changes into the new base XSLT files
• Test CDA rendering with sample documents to verify correct output
• Document all custom XSLT changes for future upgrades

Common Recompilation Issues

• Deprecated methods: The new version may deprecate methods used in custom code
• API signature changes: Method parameters or return types may have changed
• New required properties: Classes may have new required properties that must be set
• Namespace conflicts: Custom class names may conflict with new system classes

Testing After Recompilation

• Run unit tests for all custom code
• Test integration points between custom code and UCR framework
• Verify that custom business operations and services function correctly
• Test CDA transforms with representative clinical documents
• Validate any custom reports or analytics

---

Overview

After all post-upgrade verification and recompilation steps are complete, productions must be restarted across the federation. This must follow the correct startup sequence and include additional steps for Task Manager resumption and any required SAML migration.

Resuming Task Manager

Before restarting productions:

• Resume the Task Manager on each instance (it may have been suspended during upgrade)
• Verify scheduled tasks are properly configured for the new version
• Check that task schedules have not been altered by the upgrade
• Ensure critical maintenance tasks (purge, cleanup, synchronization) are scheduled

Production Restart Sequence

Follow the standard startup order: 1. Registry production: Start first. Verify MPI, consent, and registry services are operational. 2. Audit Edge production: Start to enable audit event collection. 3. Edge Gateway productions: Start each Edge Gateway. Verify registration with the Registry. 4. Access Gateway productions: Start after Edge Gateways are confirmed running. Verify Clinical Viewer access. 5. Bus production: Start if applicable. Verify external system connectivity. 6. ODS production: Start last. Verify FHIR server and data synchronization.

SAML Migration

If the upgrade crosses versions with SAML (Security Assertion Markup Language) changes:

• Run the SAML migration utility provided for the target version
• Update SAML token configurations on all components
• Verify SAML-based authentication works correctly
• Test single sign-on flows through the Clinical Viewer

Gradual Restart Approach

For large federations, consider:

• Starting one Edge Gateway first as a canary to verify upgrade success
• Monitoring for errors before starting remaining gateways
• Re-enabling data feeds from source systems incrementally
• Gradually increasing load to detect performance issues early

---

Overview

After productions are restarted and verified, several cleanup tasks must be performed to optimize the upgraded system. These tasks address data structures and indexes that may need rebuilding or optimization for the new version.

ODS Index Rebuilding

The ODS may require index rebuilding after upgrade:

• New indexes may have been added in the target version
• Existing index structures may have changed
• Run the index rebuild utility for the ODS namespace
• Monitor the rebuild process (can be time-consuming for large datasets)
• Verify query performance after index rebuilding completes

FHIR Repair Utility

The FHIR repair utility addresses potential inconsistencies:

• Run the utility to check FHIR resource integrity
• Repair any resources that do not conform to the new version's FHIR implementation
• Verify FHIR server responses after repair
• Check that FHIR search operations return correct results

BI Cube Building

Business Intelligence cubes may need rebuilding:

• Cube definitions may have changed in the new version
• New dimensions or measures may be available
• Rebuild all BI cubes used for analytics and reporting
• Verify dashboard and report accuracy after cube rebuilding
• Schedule cube rebuild tasks for ongoing maintenance

Stream Compression

Stream compression optimizes storage:

• Run the stream compression utility to compress stored streams
• This reduces disk space usage, especially for large clinical documents
• Schedule stream compression during low-usage periods as it can be resource-intensive
• Monitor disk space recovery after compression completes

General Cleanup

Additional cleanup tasks:

• Remove upgrade installer files and temporary directories
• Clean up any backup files created during the upgrade process (after confirming success)
• Review and clean up error logs from the upgrade process
• Update documentation to reflect the new version
• Notify stakeholders that the upgrade is complete

---

Overview

During and after a UCR upgrade, various errors and warnings may appear in logs and on the console. Not all of these indicate actual problems. InterSystems documents known safe-to-ignore errors for each version upgrade, and administrators must be able to distinguish these from genuine issues.

Types of Safe-to-Ignore Errors

Common categories of safe-to-ignore errors include:

• Compilation warnings for deprecated syntax: The new compiler may warn about syntax that is still functional but deprecated
• Missing class references during recompilation: Temporary errors that resolve when all classes are compiled
• Configuration migration messages: Informational messages about automatic configuration updates
• Database structure update notices: Messages about automatic schema migrations
• Temporary connectivity errors: Brief connection failures during component restarts

How to Identify Safe-to-Ignore Errors

• Consult the release notes for the target version, which list known safe-to-ignore errors
• Check InterSystems support documentation for the specific upgrade path
• Compare errors against the documented list of expected messages
• Contact InterSystems support if an error is not on the list and its severity is unclear

When to Investigate Further

Errors that should NOT be ignored include:

• Database corruption or integrity errors
• License key validation failures
• SSL/TLS certificate errors preventing inter-component communication
• Production startup failures
• Persistent compilation errors in core UCR classes
• Data loss or missing records
• Authentication or authorization failures

Best Practices

• Review all errors systematically, even if most are expected to be benign
• Document all errors encountered during the upgrade with their resolution status
• Keep a copy of the error logs for future reference
• If in doubt about an error, escalate to InterSystems support rather than ignoring it

---

Overview

After completing all upgrade steps, a comprehensive verification process ensures the upgrade was successful across the entire federation. This verification covers technical checks, functional testing, and stakeholder confirmation.

Log Review

Review logs on every upgraded instance:

• Examine the upgrade installer logs for errors or warnings
• Review the IRIS messages log for post-upgrade issues
• Check production event logs for startup errors
• Review application-specific logs for functional errors
• Compare errors found against the safe-to-ignore list

Version Confirmation

On each instance, verify:

• The Management Portal reports the correct new version
• The IRIS for Health version matches the target version
• UCR-specific version information is correct
• All system databases have been updated to the new version

Functional Testing

Perform end-to-end testing of core UCR workflows:

• Patient search: Search for patients through the Access Gateway Clinical Viewer
• Data retrieval: View clinical data from Edge Gateways through the Access Gateway
• Patient registration: Submit a new patient identity through an Edge Gateway and verify MPI processing
• Document submission: Submit a clinical document and verify it appears in the consolidated view
• Consent enforcement: Verify that consent policies are correctly applied
• Audit trail: Verify that audit events are being recorded at the Audit Edge
• FHIR access: Test FHIR API queries on the ODS (if applicable)

Performance Validation

Compare post-upgrade performance with pre-upgrade baselines:

• Patient search response times
• Clinical data retrieval latency
• Document rendering speed
• MPI matching throughput
• FHIR API response times

Stakeholder Communication

After verification:

• Notify clinical users that the system is available
• Provide a summary of changes and new features
• Document any known issues or temporary limitations
• Schedule follow-up monitoring for the first days after upgrade
• Obtain formal signoff from technical and clinical stakeholders

---