T13.2: Interprets Messaging

Overview

UCR uses a set of well-defined message types for communication between its components. Each message type serves a specific purpose in the overall data flow. Understanding these message types, their purpose, and which components send and receive them is essential for interpreting message traffic and troubleshooting communication issues.

Knowing the message types also helps when reading Visual Trace diagrams and Message Viewer results, as the message type identifies the purpose of each communication.

AddUpdateHub Messages

AddUpdateHub is one of the most common message types in UCR:

• Purpose: Register a new patient or update patient demographics at the Hub
• Source: Edge Gateway
• Target: Hub (Registry)
• When sent: Every time a new patient is encountered or patient demographics change at an Edge Gateway
• Content: Patient demographic information (name, DOB, identifiers, address, etc.)
• Result: The Hub's MPI processes the demographics for patient matching and identity linkage

QueryForData Messages

QueryForData messages drive the clinical data retrieval process:

• Purpose: Request clinical data for a specific patient
• Source: Access Gateway
• Target: Edge Gateway(s)
• When sent: When a clinician searches for or views a patient in the Clinical Viewer
• Content: Patient identifier, requested data categories, date ranges
• Result: The Edge Gateway retrieves SDA data from its ECR and returns it

InfoMessage

InfoMessages provide push-based notifications:

• Purpose: Notify components of events (new data available, patient updates)
• Source: Various (Edge Gateway, Hub)
• Target: Various (Access Gateway, other Edge Gateways)
• When sent: When an event occurs that other components need to know about
• Content: Event type and relevant context information

PIXv3 Messages

PIXv3 (Patient Identifier Cross-Reference v3) messages follow the IHE profile:

• Purpose: Query and update patient identity cross-references in the MPI
• Source: Edge Gateways, Access Gateways
• Target: Hub (Registry)
• When sent: When resolving patient identities across facilities
• Content: Patient identifiers and demographic matching criteria

XDS.b Messages

XDS.b (Cross-Enterprise Document Sharing) messages follow the IHE profile:

• Purpose: Register, query, and retrieve clinical documents
• Source/Target: Between Edge Gateways and Hub
• When sent: When documents are registered or retrieved
• Content: Document metadata, document references, document content

XCA Messages

XCA (Cross-Community Access) messages enable cross-federation queries:

• Purpose: Query for and retrieve data across community boundaries
• Source/Target: Between UCR federations or with external communities
• When sent: When data is needed from outside the local federation
• Content: Patient identity and data queries conforming to IHE XCA profile

Message Flow Summary

| Message Type | From | To | Purpose | |---|---|---|---| | AddUpdateHub | Edge Gateway | Hub | Patient registration/update | | QueryForData | Access Gateway | Edge Gateway | Clinical data retrieval | | InfoMessage | Various | Various | Event notifications | | PIXv3 | Edge/Access Gateway | Hub | Patient identity resolution | | XDS.b | Edge Gateway | Hub | Document registration/retrieval | | XCA | Access Gateway | External | Cross-community queries |

---

Overview

All UCR inter-component communication uses SOAP (Simple Object Access Protocol) as the messaging framework. SOAP provides a standardized, extensible message format that supports the security requirements of healthcare data exchange. Understanding the SOAP message structure helps when examining message content in the Message Viewer and diagnosing communication failures.

UCR's use of SOAP aligns with IHE (Integrating the Healthcare Enterprise) profiles, which specify SOAP-based web services for healthcare interoperability.

SOAP Envelope Structure

A SOAP message has three main parts: ``` ```

SOAP Header Contents

The SOAP header in UCR messages typically contains:

• WS-Security header: Includes security tokens, timestamps, and digital signatures
• SAML assertions: Identity assertions for the sending component or user
• Addressing information: WS-Addressing elements specifying the target endpoint and action
• Custom headers: Any additional routing or processing metadata

SOAP Body Contents

The SOAP body contains the actual message content, which varies by message type:

• For AddUpdateHub: Patient demographic data in the UCR schema
• For QueryForData: Query parameters including patient identifier and data categories
• For PIXv3: Patient identifier cross-reference query or response data
• For XDS.b: Document metadata or document content

WS-Security

WS-Security provides message-level security independent of the transport layer:

• Authentication: Verifies the identity of the sending component
• Message integrity: Digital signatures ensure messages are not tampered with in transit
• Timestamps: Prevent replay attacks by validating message freshness
• SAML tokens: Carry identity assertions for federated authentication

SSL/TLS Transport Security

In addition to message-level WS-Security, all UCR communications use SSL/TLS:

• Encrypts the entire communication channel between components
• Requires SSL/TLS configurations on each UCR instance
• Certificate management is essential for maintaining secure communications
• Both client and server certificates may be required depending on configuration

Credential Configuration

Each business operation that communicates with another UCR component must be configured with:

• The target endpoint URL (HTTPS)
• SSL/TLS configuration reference
• Credentials (username/password or certificate-based)
• WS-Security settings appropriate for the target component

---

Overview

A UCR production contains both standard business hosts (shipped with the product and maintained by InterSystems) and custom code (added during implementation to meet facility-specific requirements). Distinguishing between the two is critical for troubleshooting, because standard components are extensively tested and rarely the source of issues, while custom code is more likely to contain bugs or configuration errors.

When troubleshooting a problem, the general rule is to check custom code first before suspecting standard UCR components.

Standard UCR Business Hosts

Standard business hosts are part of the UCR product installation:

• They implement core UCR functionality (SDA conversion, MPI communication, ECR storage, data retrieval)
• They follow predictable patterns documented by InterSystems
• They are updated with product patches and upgrades
• They should not be modified directly (modifications would be overwritten during upgrades)
• Examples include the standard AddUpdateHub operation, the MPI matching process, and the ECR storage operation

Custom Code Locations

Custom code in UCR productions is typically found in these locations:

• PreProcessor: A custom business process or rule that runs before the standard processing chain. Used to filter, enrich, or modify inbound data before it enters the standard flow
• PostProcessor: A custom business process or rule that runs after the standard processing chain. Used for additional routing, notifications, or data extraction after standard processing completes
• Custom DTL transformations: Data Transformation Language transformations that map facility-specific data formats to SDA. These handle the unique aspects of each source system's data
• Custom business processes: Additional orchestration logic for facility-specific workflows
• Custom business operations: Additional operations for integrating with facility-specific systems

Identifying Custom vs Standard Components

In the Production Configuration page:

• Standard components typically have class names starting with `HS.` (HealthShare namespace)
• Custom components typically have class names in a facility-specific namespace
• Custom components may be marked with comments or naming conventions indicating they are custom
• The production documentation should identify which components are custom additions

Troubleshooting Implications

When diagnosing issues: 1. Check custom code first: Custom PreProcessors, PostProcessors, and DTL transformations are the most common source of data flow issues 2. Verify custom DTL mappings: Facility-specific transformations may have mapping errors or missing fields 3. Review PreProcessor logic: Custom filtering or routing logic may be incorrectly discarding or redirecting messages 4. Check PostProcessor errors: Additional processing after the standard flow may fail without affecting the core data flow 5. Standard components last: Only investigate standard UCR components after ruling out custom code issues

---

Overview

Effective troubleshooting of UCR messaging issues requires knowing where to start looking. The Management Portal provides several complementary tools, each offering a different perspective on the production's behavior. Using these tools in the right order saves time and leads to faster resolution.

The recommended systematic approach is: start broad (Event Log for errors), then narrow down (Visual Trace for specific messages), then check infrastructure (Production Monitor for component health).

Production Event Log — First Stop

The Event Log should be the first place to check when troubleshooting:

• What it shows: Errors, warnings, and informational messages generated by business hosts during processing
• Why start here: It gives an immediate overview of what is going wrong and which components are affected
• How to use: Filter by time range and severity (start with Errors, then Warnings)
• Key information: Error text, component name, timestamp, and stack trace for detailed diagnosis
• Quick search: Search by text string to find specific error patterns

Visual Trace — Follow the Message

After identifying an issue in the Event Log, use Visual Trace to follow specific messages:

• What it shows: The complete path of a message through the production, including all components that processed it
• Why use it: It reveals exactly where in the processing chain a message failed or was lost
• How to use: Find the message in the Message Viewer, then click the Visual Trace link
• Key information: Message flow diagram, timing, status at each step, and error details

Message Viewer — Search and Browse

The Message Viewer is essential for finding specific messages:

• What it shows: All messages processed by the production, with filtering and search capabilities
• Why use it: To find specific messages by patient, time, source, target, or status
• How to use: Apply filters to narrow down to the relevant messages, then examine headers and content
• Key information: Message headers, body content, session grouping, and links to Visual Trace

Production Monitor — Component Health

The Production Monitor provides a high-level view of production health:

• What it shows: Status of each component (Running, Stopped, Error, Disabled), queue depths, error counts
• Why use it: To identify systemic issues (stopped components, growing queues, repeated errors)
• How to use: Review the status of all components, paying attention to error counts and queue depths
• Key information: Component status, last activity time, queue depth, error count

Systematic Troubleshooting Workflow

1. Event Log: Check for recent errors to identify the affected component and error type 2. Message Viewer: Find the specific messages related to the reported issue 3. Visual Trace: Follow the problematic message through the production to identify the failure point 4. Production Monitor: Check if the issue is isolated or systemic (component down, growing queues) 5. Detailed investigation: Examine message content, check configuration, review custom code

---

Overview

Every message in a UCR production carries a set of header fields that provide metadata about the message's origin, destination, timing, and status. Understanding these header fields is essential for finding specific messages, tracing related messages, and diagnosing timing and processing issues.

The Message Viewer displays header fields for every message and allows filtering on most of them, making header knowledge directly applicable to troubleshooting workflows.

Source and Target

• Source: Identifies the business host that created the message. This is the component that initiated the communication
• Target: Identifies the business host that should receive and process the message
• Usage: Filtering by Source and Target helps narrow down messages to specific communication paths (e.g., all messages from the inbound HL7 service to the SDA conversion process)

SessionId

• Purpose: Groups all messages that result from a single inbound event into a logical session
• Example: When an HL7 message arrives at the Edge Gateway, it generates multiple internal messages (routing, transformation, storage, Hub notification). All of these share the same SessionId
• Usage: Filtering by SessionId shows the complete processing chain for a single inbound event. This is especially useful when a Visual Trace shows a complex flow and you need to see all related messages

MessageId

• Purpose: Uniquely identifies each individual message within the production
• Usage: Use MessageId to reference a specific message when documenting issues or communicating with support
• Uniqueness: Each message has a unique MessageId, even within the same session

CorrelationId

• Purpose: Links request and response messages together
• Example: When the Access Gateway sends a QueryForData request to an Edge Gateway, the response message will have a CorrelationId matching the request's MessageId
• Usage: Use CorrelationId to find the response to a specific request, or to find the request that generated a specific response

TimeCreated and TimeProcessed

• TimeCreated: Timestamp when the message was created by the source component
• TimeProcessed: Timestamp when the message was fully processed by the target component
• Usage: The difference between these two timestamps indicates processing time. Long gaps may indicate performance issues, queue backlogs, or component problems
• Time range filtering: The Message Viewer allows filtering by time range to find messages from specific periods

Status

• Completed: The message was successfully processed by the target component
• Error: Processing failed; check the Event Log and Visual Trace for error details
• Suspended: Processing was halted and the message is awaiting manual intervention
• Queued: The message is waiting in the target component's queue for processing
• Discarded: The message was deliberately dropped during processing

Viewing Message Body Content

In addition to headers, the Message Viewer allows viewing the full message body:

• SDA content: Clinical data in SDA XML format
• SOAP messages: Full SOAP envelope for inter-component communications
• HL7 messages: HL7v2 segments for inbound data feeds
• Other formats: CDA, CSV, or any other format used by the production

---

Overview

In a UCR production, a single inbound event (such as receiving an HL7 message from a source system) typically generates multiple internal messages as the data is routed, transformed, stored, and communicated to other components. All of these internal messages are grouped into a session identified by a shared SessionId. Understanding sessions is essential for tracing the complete processing chain of an inbound event.

Sessions provide a logical grouping that makes it possible to see the full impact of a single inbound message, rather than looking at each internal message in isolation.

How Sessions Work

1. An inbound business service receives a message from an external system 2. The production assigns a SessionId to this inbound event 3. Every subsequent message generated as a result of processing this inbound event inherits the same SessionId 4. The session encompasses all internal routing, transformation, storage, and notification messages

Session Scope

A session exists within a single production:

• All messages within the Edge Gateway production that result from one inbound message share a SessionId
• When the Edge Gateway sends a message to the Hub (a different production), the Hub creates its own session for processing that message
• Cross-production tracing requires correlating sessions across productions using message content or timing

Using Sessions for Troubleshooting

Sessions are invaluable for troubleshooting because they show the complete processing chain:

• Finding all related messages: Filter the Message Viewer by SessionId to see every message generated from a single inbound event
• Identifying incomplete processing: If a session has fewer messages than expected, a processing step may have failed
• Timing analysis: Compare timestamps across session messages to identify slow processing steps
• Error correlation: If one message in a session has an error, the SessionId helps find all related messages to understand the context

Session View in Message Viewer

The Message Viewer provides a session view:

• Click on a message's SessionId link to see all messages in that session
• Messages are displayed in chronological order
• Each message shows its Source, Target, Status, and timestamps
• Visual Trace can be accessed for any message in the session to see the graphical flow

Practical Example

When an HL7 ADT message arrives at an Edge Gateway, a typical session might include: 1. Inbound HL7 message received by the TCP business service 2. Message routed to the HL7-to-SDA conversion process 3. SDA streamlets sent to the ECR storage operation 4. Patient demographics sent to the AddUpdateHub operation 5. AddUpdateHub SOAP message sent to the Hub 6. Response from the Hub received and processed

All six of these messages share the same SessionId, making it easy to follow the complete flow.

---

Overview

Every message in a UCR production has a status value that indicates its current processing state. Understanding these status values is fundamental to troubleshooting, as they immediately tell you whether a message was processed successfully, failed, is waiting for attention, or was intentionally dropped. The Message Viewer displays the status for every message, and filtering by status is one of the most common troubleshooting techniques.

Completed

• Meaning: The message was successfully received by the target component and processed without errors
• Action required: None — this is the expected status for normal operation
• Troubleshooting note: Even Completed messages may have produced unexpected results; if the output is wrong but the status is Completed, the issue may be in the transformation logic or configuration rather than the processing pipeline

Error

• Meaning: The target component attempted to process the message but encountered a failure
• Action required: Investigate the error to determine root cause and fix the issue
• Troubleshooting steps:
• Check the Event Log for error details and stack traces
• View the Visual Trace to see exactly where processing failed
• Examine the message content to verify it is well-formed
• Check the target component's configuration for issues
• Common causes: Malformed message content, connection failures to external systems, configuration errors, custom code bugs

Suspended

• Meaning: Processing was halted and the message is awaiting manual intervention. The message is not lost — it remains in the queue but is not being processed
• Action required: Review the message and decide whether to resubmit, edit and resubmit, or discard
• How messages get suspended:
• A business host's retry count was exceeded
• An administrator manually suspended the message
• A business rule or process explicitly suspended the message
• Resolution: Use the Resend Editor to resubmit the message after fixing the underlying issue
• Options in Resend Editor: Send to original target, send to a different target, edit message before resending

Queued

• Meaning: The message is waiting in the target component's processing queue and has not yet been processed
• Action required: Usually no immediate action — messages in the queue will be processed as the component becomes available
• When to investigate:
• If the queue is growing and messages are not being processed (component may be stopped or overloaded)
• If the queue depth exceeds the QueueCountAlert threshold
• If messages have been queued for an unusually long time
• Common causes of persistent queuing: Target component is stopped, component is overloaded, PoolSize is too low for the workload

Discarded

• Meaning: The message was deliberately dropped during processing, typically by a business rule or process that determined the message should not be processed further
• Action required: Verify that discarding was intentional and correct
• When to investigate: If messages are being discarded unexpectedly, check business rules and routing logic for errors
• Note: Discarded messages are still visible in the Message Viewer for audit purposes

Status-Based Troubleshooting Summary

| Status | Priority | First Action | |---|---|---| | Error | High | Check Event Log for error details | | Suspended | High | Review in Resend Editor; fix and resubmit | | Queued (growing) | Medium | Check target component status and PoolSize | | Discarded (unexpected) | Medium | Review business rules and routing logic | | Completed (wrong result) | Low | Review transformation and configuration |

---