T13.1: Traces Patient Data Flow

Overview

The Testing Service is a built-in Management Portal page that allows administrators and developers to send test messages directly into a production's business services. This is essential for verifying that UCR productions are correctly configured before connecting live data sources. It provides a controlled way to inject sample data and observe how it flows through the production.

In a UCR context, testing typically begins at the Edge Gateway by sending sample clinical messages (HL7, CDA, or SDA) through the inbound business service, then verifying the message is processed correctly through the production chain.

Accessing the Testing Service

The Testing Service is accessed from the Management Portal under the production's namespace:

• Navigate to Interoperability > Test > Testing Service
• Select the business service to test
• Provide the test message content in the appropriate format
• Submit the message and observe the response

Testing Edge Gateway Productions

When testing the Edge Gateway production:

• Prepare sample inbound messages in the format expected by the facility's data feed (HL7v2, CDA, etc.)
• Send the message to the appropriate inbound business service
• Verify the message is received, transformed to SDA, and stored in the Edge Cache Repository
• Check the Visual Trace to confirm the message flowed through all expected components
• Verify patient identity information was submitted to the Registry

Testing Hub Productions

When testing the Hub (Registry) production:

• Send sample AddUpdateHub messages to verify patient registration processing
• Test PIXv3 or patient identity queries to verify MPI resolution
• Verify consent policy enforcement by testing with different patient consent scenarios

End-to-End Verification

A complete end-to-end test involves: 1. Sending a sample message into the Edge Gateway 2. Verifying the data is stored in the ECR 3. Confirming the patient identity reaches the Registry's MPI 4. Using the Access Gateway or Clinical Viewer to query for the patient 5. Verifying the clinical data is retrieved and displayed correctly

---

Overview

Each UCR component (Edge Gateway, Hub, Access Gateway) runs its own production containing business hosts. Understanding which business hosts exist in each production and their roles in the data flow is critical for tracing patient data and troubleshooting issues. Business hosts are categorized as business services (inbound), business processes (routing and transformation), and business operations (outbound and storage).

Edge Gateway Production Components

The Edge Gateway production typically contains:

• Inbound business services: Receive data feeds from facility source systems (e.g., HL7 TCP listeners, file services, HTTP services)
• SDA conversion processes: Transform inbound data formats into SDA (Summary Document Architecture)
• ECR storage operations: Store SDA streamlets in the Edge Cache Repository
• Hub notification operations: Send patient registration (AddUpdateHub) and document notification messages to the Registry
• Translation and routing processes: Apply translation profiles and route messages to appropriate targets

Hub (Registry) Production Components

The Hub production typically contains:

• Inbound business services: Receive messages from Edge Gateways and Access Gateways
• MPI processes: Handle patient identity matching and linkage
• Consent management processes: Evaluate and enforce consent policies
• Registry operations: Manage patient, clinician, and system registries
• Query response operations: Respond to patient identity and data location queries

Access Gateway Production Components

The Access Gateway production typically contains:

• Clinical Viewer services: Handle user requests from the Clinical Viewer web application
• Query processes: Orchestrate data retrieval from multiple Edge Gateways
• Hub communication operations: Query the Hub for patient identity and data location information
• Edge Gateway communication operations: Fetch clinical data from Edge Gateways
• Data aggregation processes: Combine data from multiple sources into a unified view
• Consent filtering processes: Apply consent policies to filter retrieved data

---

Overview

Visual Trace is one of the most powerful troubleshooting tools in the Management Portal. It provides a graphical representation of how a message flows through the production, showing every component that touched the message, the sequence of processing, and the time taken at each step. For UCR data flow tracing, Visual Trace is the primary tool for understanding what happened to a specific patient message.

Visual Trace is accessed from the Management Portal under Interoperability > View > Messages, then clicking the Visual Trace link for a specific message session.

Reading the Trace Diagram

The Visual Trace diagram uses a standard notation:

• Vertical lines: Each vertical line represents a business host (service, process, or operation) in the production
• Horizontal arrows: Arrows between vertical lines show messages flowing from one component to another
• Arrow direction: Left-to-right arrows indicate request messages; right-to-left arrows indicate response messages
• Timestamps: Each arrow is annotated with the time the message was sent and the processing duration
• Color coding: Green indicates successful processing; red indicates errors; yellow indicates warnings

Expanding Trace Nodes

Clicking on any message arrow in the trace diagram expands it to show:

• The full message header information (Source, Target, MessageId, SessionId)
• The message body content (viewable in the appropriate format)
• Processing timestamps (TimeCreated, TimeProcessed)
• The message status (Completed, Error, etc.)

Tracing a Complete Patient Data Flow

For a typical Edge Gateway inbound flow, the Visual Trace shows: 1. Message received by the inbound business service 2. Message routed to the SDA conversion process 3. Transformed SDA sent to the ECR storage operation 4. Patient identity notification sent to the Hub notification operation 5. Hub acknowledgment received back

Each step is clearly visible in the trace, making it easy to identify where data stops flowing if there is a problem.

Tips for Effective Visual Trace Usage

• Start from the inbound business service to see the complete message journey
• Use the session view to see all related messages from a single inbound event
• Compare traces of successful and failed messages to identify differences
• Note the timing at each step to identify performance bottlenecks
• Use Visual Trace in combination with the Event Log for complete troubleshooting

---

Overview

Data undergoes several transformations as it flows through the UCR architecture. Understanding where these transformations occur is essential for tracing data and diagnosing issues where data appears incorrect or incomplete. The key transformation is the conversion of source data into SDA format at the Edge Gateway, which normalizes data from disparate source systems into a common clinical data model.

Transformation at the Edge Gateway (Inbound)

The most significant transformation occurs when data enters the Edge Gateway:

• Source format parsing: Inbound data arrives in its native format (HL7v2, CDA, CSV, etc.)
• SDA conversion: A conversion process transforms the parsed data into SDA (Summary Document Architecture) format
• Translation profiles: During conversion, translation profiles map source-specific values (local codes, identifiers, terminology) to standardized values
• Data normalization: Dates, names, identifiers, and coded values are normalized to a consistent format

The SDA format is InterSystems' canonical clinical data model. It represents clinical data as a set of streamlets, each representing a clinical concept (diagnosis, medication, lab result, allergy, etc.).

Storage in the Edge Cache Repository

After transformation to SDA:

• SDA streamlets are stored in the ECR indexed by patient identifier
• Each streamlet type is stored separately (e.g., all medications for a patient are in a medications streamlet)
• The ECR maintains the full history of all data received for each patient
• Streamlets include provenance information identifying the source system and time of receipt

Translation Profiles

Translation profiles are configured at the Edge Gateway to handle facility-specific data mapping:

• Map local code systems to standard terminologies (SNOMED, LOINC, ICD-10)
• Translate facility-specific identifiers to federation-standard formats
• Handle facility-specific data formatting conventions
• Multiple translation profiles can be applied in sequence

Transformation at the Access Gateway (Retrieval)

When data is retrieved for display in the Clinical Viewer:

• The Access Gateway requests SDA data from Edge Gateways
• Data from multiple Edge Gateways is aggregated into a unified patient view
• Consent policies may filter out certain data categories
• SDA is rendered into the Clinical Viewer display format

---

Overview

The Edge Cache Repository (ECR) is the local clinical data store at each Edge Gateway. It is the definitive repository for all clinical data collected from a facility's source systems. Understanding the flow of data into and within the ECR is essential for troubleshooting data availability issues and for tracing patient data through the UCR system.

Every piece of clinical data that enters an Edge Gateway is ultimately stored in the ECR as SDA streamlets. The ECR is not a pass-through; it is a persistent store that retains data for retrieval by Access Gateways.

Inbound Flow to ECR

The typical flow from inbound message to ECR storage follows these steps: 1. Message reception: An inbound business service receives a message from a source system (HL7 feed, CDA document, etc.) 2. Message routing: The message is routed to the appropriate processing chain based on source type and message type 3. Data parsing: The message is parsed from its native format into an intermediate representation 4. SDA conversion: The parsed data is converted into SDA streamlets 5. Translation: Translation profiles are applied to standardize coded values and identifiers 6. ECR storage: The resulting SDA streamlets are stored in the ECR, indexed by the patient's facility identifier

SDA Streamlet Organization

Data in the ECR is organized as SDA streamlets, each representing a clinical concept:

• Patient demographics: Name, address, date of birth, identifiers
• Encounters: Hospital visits, office visits, emergency department visits
• Diagnoses: Active and historical diagnoses with codes
• Medications: Current and historical medication orders
• Lab results: Laboratory test results with values and reference ranges
• Allergies: Known allergies and adverse reactions
• Procedures: Surgical and diagnostic procedures
• Documents: Clinical documents (notes, reports, discharge summaries)

Each streamlet type is stored separately, allowing efficient retrieval of specific data categories.

Patient Indexing in the ECR

ECR data is indexed by the patient's local facility identifier (typically the MRN). When data arrives:

• The patient is identified by the identifier in the inbound message
• If the patient already exists in the ECR, new data is added to existing streamlets
• If the patient is new, a new patient record is created in the ECR
• The patient's identity is also registered with the Hub's MPI for cross-facility linkage

ECR Data Persistence

Data stored in the ECR persists until explicitly purged. This means:

• Historical data accumulates over time
• The ECR grows with each data feed
• Purging policies must be established to manage storage
• Purging must be coordinated with the Hub to maintain data consistency

---

Overview

The data retrieval flow is the reverse of the ingestion flow. When a clinician uses the Clinical Viewer to look up a patient, a complex multi-step process occurs behind the scenes: the Access Gateway communicates with the Hub to identify the patient and locate their data, then queries the relevant Edge Gateways to retrieve the clinical data, applies consent policies, and presents the aggregated view to the clinician.

Understanding this flow is critical for diagnosing issues where a clinician cannot see expected patient data in the Clinical Viewer.

Step-by-Step Retrieval Flow

1. Patient search: The clinician enters a patient search in the Clinical Viewer (name, MRN, date of birth, etc.) 2. Access Gateway to Hub: The Access Gateway sends a patient identity query to the Hub's Registry 3. MPI resolution: The Hub's MPI resolves the query to a Master Patient Index ID (MPIID) and returns all known identities for that patient across all facilities 4. Data location identification: The Hub identifies which Edge Gateways hold data for this patient based on registered documents and patient identities 5. Edge Gateway queries: The Access Gateway sends clinical data queries to each relevant Edge Gateway 6. SDA retrieval: Each Edge Gateway retrieves SDA streamlets from its ECR for the requested patient and returns them to the Access Gateway 7. Data aggregation: The Access Gateway combines SDA data from all responding Edge Gateways into a unified clinical record 8. Consent application: Patient consent policies (managed by the Hub's Consent Manager) are applied to filter data the clinician is not authorized to see 9. Clinical Viewer display: The aggregated, consent-filtered data is presented in the Clinical Viewer

Key Communication Points

• Access Gateway to Hub: SOAP/HTTPS for patient identity and data location queries
• Access Gateway to Edge Gateways: SOAP/HTTPS for clinical data retrieval
• All communications secured by WS-Security and SSL/TLS
• Timeout settings on each query step can affect data completeness

Transient Data at the Access Gateway

The Access Gateway does not permanently store retrieved clinical data:

• Data is held transiently during the clinician's active session
• When the session ends, the transient data is discarded
• Each new patient lookup triggers a fresh retrieval from Edge Gateways
• This ensures clinicians always see the most current data

Performance Considerations

The retrieval flow involves multiple network round trips:

• Slow network connections to distant Edge Gateways can delay results
• Edge Gateways with large data volumes may take longer to respond
• Timeout settings must be balanced between completeness and responsiveness
• If an Edge Gateway does not respond within the timeout, its data will be missing from the display

---

Overview

Missing or incomplete patient data in the Clinical Viewer is one of the most common troubleshooting scenarios in a UCR deployment. Because data flows through multiple components (Edge Gateway, Hub, Access Gateway), the missing data could be caused by a failure at any point in the chain. A systematic approach working from inbound to outbound is essential for efficient diagnosis.

The key principle is to start at the source (Edge Gateway) and work forward through each component until the point where data stops flowing is identified.

Step 1: Check the Edge Gateway

First, verify that the data actually arrived at the Edge Gateway:

• Open the Management Portal for the Edge Gateway's namespace
• Use Visual Trace or Message Viewer to search for messages from the source system
• Look for the specific patient identifier (MRN) and time range
• If no messages are found, the issue is upstream (the source system did not send the data or the business service did not receive it)

Step 2: Verify SDA Conversion

If messages arrived but data is not in the ECR:

• Check the Visual Trace for the inbound message to see if SDA conversion completed
• Look for errors in the Event Log related to the conversion process
• Verify the translation profile is correctly configured for the source system
• Check if the SDA streamlet was successfully stored in the ECR
• Test by querying the ECR directly for the patient's data

Step 3: Check MPI Linkage

If data is in the ECR but not visible in the Clinical Viewer:

• Verify the patient's identity was registered with the Hub's MPI
• Check the Hub's Event Log for any errors in patient registration processing
• Confirm the patient's MPIID links the correct identities across facilities
• If the patient has multiple MRNs that are not linked, the Access Gateway may not query all relevant Edge Gateways

Step 4: Verify Consent Policies

If the patient is correctly linked but data is still missing:

• Check the patient's consent policies in the Hub's Consent Manager
• Verify that the consent policies allow the requesting clinician to see the data
• Check if specific data categories are blocked by consent (e.g., behavioral health, substance abuse)
• Test with a user who has full access to confirm whether consent is the issue

Step 5: Check Access Gateway Timeouts

If some but not all data is missing:

• Check the Access Gateway's timeout settings for Edge Gateway queries
• Review the Access Gateway's Event Log for timeout errors
• Verify network connectivity between the Access Gateway and all relevant Edge Gateways
• Check if specific Edge Gateways are slow to respond or unreachable
• Increase timeout values temporarily for testing purposes

Common Root Causes Summary

| Symptom | Likely Cause | |---------|-------------| | No data at all | Data not sent by source system or Edge Gateway not receiving | | Data in ECR but not in Clinical Viewer | MPI linkage issue or consent policy blocking | | Partial data missing | Timeout on one Edge Gateway or consent filtering specific categories | | Data was there, now missing | Purge task removed historical data | | Wrong patient data | MPI false positive linking wrong patients |

---

Overview

The Master Patient Index (MPI) is a foundational component of the UCR architecture, running as a service within the Hub (Registry). Its primary purpose is to link patient identities across different facilities so that data from multiple Edge Gateways can be aggregated into a single, consolidated patient view. Without the MPI, each Edge Gateway's data would be isolated, and clinicians would have no way to see a patient's complete record across the federation.

The MPI assigns each unique patient a Master Patient Index ID (MPIID), which serves as the cross-facility identifier that ties together all of a patient's records regardless of which facility they visited.

How the MPI Works

1. Identity registration: When an Edge Gateway receives patient data, it sends the patient's demographic information and local identifier (MRN) to the Hub 2. Matching algorithms: The Hub's MPI engine runs matching algorithms against its existing patient database 3. Match determination: The algorithms produce a confidence score; if above the threshold, the patient is linked to an existing MPIID 4. New patient creation: If no match is found, a new MPIID is created for this patient 5. Identity linking: The patient's facility-specific identifier (MRN) is linked to the MPIID

Matching Algorithms

The MPI uses configurable matching algorithms that consider:

• Deterministic matching: Exact match on specific fields (e.g., SSN, national ID)
• Probabilistic matching: Weighted comparison of multiple demographic fields (name, date of birth, address, phone number, gender)
• Threshold configuration: Administrators set confidence score thresholds for automatic linking vs. manual review
• Blocking strategies: Reduce the search space by grouping potential matches before detailed comparison

MPIID and Data Aggregation

The MPIID is the key that enables data aggregation:

• When a clinician searches for a patient in the Clinical Viewer, the Access Gateway queries the MPI
• The MPI returns the patient's MPIID and all linked facility identifiers
• The Access Gateway then queries each relevant Edge Gateway using the facility-specific identifier
• Data from all responding Edge Gateways is combined into a unified view under the single MPIID

Impact of MPI Errors

MPI errors have direct consequences for data visibility:

• False positives (incorrect matches): Two different patients linked to the same MPIID, resulting in mixed patient data in the Clinical Viewer — a patient safety risk
• False negatives (missed matches): The same patient has multiple unlinked MPIIDs, resulting in fragmented data — the clinician cannot see the complete patient record
• Duplicate MPIIDs: A patient has multiple MPIIDs due to variations in demographic data across facilities

MPI Maintenance

Ongoing MPI management includes:

• Reviewing potential matches flagged for manual review
• Resolving known duplicates by merging MPIIDs
• Splitting incorrectly linked patients
• Tuning matching algorithm thresholds based on match quality metrics
• Monitoring match rates and false positive/negative rates

---