T2.7: Manages Messages

Overview

Manual message purging provides administrators with direct control over removing old messages from the InterSystems HealthConnect database. As productions process thousands or millions of messages over time, message history accumulates and can consume significant database space. Regular purging is essential for database maintenance and performance. The manual purge interface is accessed through the Production menu, navigating to the Message Viewer and selecting the Purge option.

Session-Based Purging

The purge operation is organized around message sessions rather than individual messages. A session represents the complete processing of a message through the production, including the original message received by a Business Service, any routing through Business Processes, sending through Business Operations, and all acknowledgments and responses. Purging a session removes all messages and related data associated with that complete message flow.

Namespace Isolation

Purge operations are namespace-specific. When you perform a manual purge, it only affects messages in the current namespace where the production runs. If you have multiple productions in different namespaces, you must purge each namespace separately. This namespace isolation prevents accidental purging of messages from unrelated productions and provides granular control over message retention by production.

Configuration Options

The manual purge interface provides several configuration options to control what gets purged and how. Understanding these options is essential for safe and effective message purging. The three main options - Include Message Bodies, Purge Only Completed Sessions, and retention period settings - each affect the purge behavior in important ways covered in the following sections.

Purpose of the Setting

The Include Message Bodies option is one of the most important purge configuration settings and should almost always be set to On. This setting controls whether the purge operation deletes just the message header records or both headers and message body content. Message headers contain metadata like timestamps, source and target components, session IDs, and message status, while message bodies contain the actual HL7 message content.

Setting to On (Recommended)

When Include Message Bodies is set to On, the purge operation deletes both header records and body records, completely removing all traces of the purged messages from the database. This is the correct setting for normal purge operations because it ensures complete cleanup and frees maximum database space. The message metadata is removed along with the message content, and database storage is reclaimed.

Setting to Off (Caution)

If Include Message Bodies is set to Off, the purge operation deletes only header records while leaving message body records in the database. This creates orphaned records - message bodies that no longer have corresponding headers and cannot be accessed or viewed through the message trace. These orphaned records consume database space but provide no value since they can't be correlated with message flow. This setting should be Off only in rare specialized scenarios where message content must be retained for compliance reasons even after purging message flow metadata.

Exam Importance

The Include Message Bodies setting is critical enough that it's frequently tested on the certification exam. Remember that the correct setting for normal purge operations is On. Orphaned message body records are a common database maintenance problem caused by incorrect purge configuration, and understanding the purpose and recommended setting of Include Message Bodies helps prevent this issue.

Purpose of the Setting

The Purge Only Completed Sessions option controls whether the purge operation includes additional validation to ensure only completed message sessions are purged. This setting involves a trade-off between absolute safety and purge performance. Understanding when to use each setting is important for effective database maintenance.

Setting to On (Safe Mode)

When Purge Only Completed Sessions is set to On, the purge operation performs additional checks to verify that each message session is completely finished before purging it. A session is considered completed when all messages in the session have reached terminal states (completed, errored, or discarded) and no messages are still in progress or waiting for responses. This setting provides guaranteed safety - you will never accidentally purge a message that's still being processed or waiting for an acknowledgment.

Performance Trade-off

The safety of Purge Only Completed Sessions = On comes at a cost: increased CPU usage and slower purge performance. The additional validation requires examining all messages in each session to verify completion status, which involves more database queries and processing. For large purge operations affecting millions of message sessions, this validation overhead can be significant, causing the purge to take hours longer than it would without this checking.

Setting to Off (Fast Mode)

When Purge Only Completed Sessions is set to Off, the purge operation skips the completion validation and purges all sessions that meet the age criteria, regardless of whether they're completed. This is much faster because it avoids the validation overhead. However, it carries a small risk - if a message session is somehow still in progress despite being old enough to meet the purge criteria, it could be purged while still active.

Practical Recommendations

The practical recommendation is to use Purge Only Completed Sessions = On when purging recent messages (perhaps messages from the last week or month) to ensure no active sessions are inadvertently purged. Use Purge Only Completed Sessions = Off when purging very old messages (perhaps messages over 90 days old) where you can be confident no sessions are still active. The speed improvement for large historical purges justifies the minimal risk.

Retention Period Configuration

The message retention period setting controls how far back in time the purge operation will go, protecting recent messages from being purged. This is configured using the "Do not purge most recent N days" parameter in the manual purge interface. Any messages newer than N days are automatically excluded from the purge, regardless of other settings. Only messages older than the retention period are eligible for purging.

Compliance Considerations

Determining the appropriate retention period involves balancing several factors. Healthcare organizations often have compliance requirements mandating minimum message retention periods. For example, regulations might require retaining messages for 90 days, 6 months, or even years depending on message type and jurisdiction. The retention period must satisfy these compliance requirements to ensure the organization doesn't purge messages that must be retained for legal or regulatory reasons.

Operational Needs

Operational needs also influence retention period decisions. Messages from the past several days or weeks are frequently needed for troubleshooting integration issues, investigating discrepancies between systems, or answering questions about when specific messages were sent. A too-short retention period creates operational problems by purging messages while they're still valuable for operations and support. Most production environments retain at least 30 days of messages for operational purposes.

Storage and Performance Balance

Storage capacity and database performance considerations affect the other end of the equation. Longer retention means more messages in the database, consuming more storage and potentially affecting query performance in the message viewer. Very high-volume productions might find that retaining a year of messages creates database size and performance challenges. The retention period must balance compliance and operational needs against storage and performance constraints.

Typical Retention Values

Typical retention periods vary by environment and message volume. Development and test environments often use short retention (7-30 days) since historical messages have limited value. Production environments typically use 30-90 days for active message retention, sufficient for most operational needs without excessive storage. Some organizations implement tiered retention, keeping recent messages in the operational database and archiving older messages to separate storage for long-term compliance retention.

Benefits of Automated Purging

While manual purging is useful for ad-hoc cleanup or troubleshooting, production environments should implement automated purging using the Task Manager to ensure regular, consistent message cleanup without manual intervention. The Task Manager in InterSystems IRIS allows you to schedule recurring tasks, including message purge tasks, that run automatically according to a defined schedule.

Creating Purge Tasks

Automated purge tasks are created through the System Management Portal, navigating to System Operation, then Task Manager, then New Task. When creating a purge task, the critical configuration is specifying the task class name: Ens.Util.Task.Purge. This class name is important to know for the certification exam - questions may ask you to identify the correct class for automated purging or troubleshoot a purge task that's using the wrong class.

Task Configuration Options

The purge task configuration includes all the same options available in manual purging - Include Message Bodies, Purge Only Completed Sessions, retention period, and type of data to purge (covered in the next section). These parameters are set in the task definition and apply each time the task runs. This ensures consistent purge behavior - the same retention period and settings are applied automatically on the defined schedule.

Scheduling Options

Task scheduling provides flexibility for when purge operations run. Most production environments schedule purge tasks to run daily during off-peak hours, typically late at night or early morning when message volume is low and system resources are available. Weekly purging is appropriate for lower-volume environments or when longer retention periods mean there's less urgency for frequent cleanup. The schedule can be configured using various patterns - specific days of the week, specific days of the month, or complex schedules using cron-like syntax.

Namespace Considerations

A critical point to understand is that purge tasks are namespace-specific. Each task runs in a specific namespace and only purges messages in that namespace. If your IRIS instance hosts multiple productions in different namespaces, you must create separate purge tasks for each namespace. Exam questions sometimes test whether you recognize that a single purge task won't purge messages across all namespaces - you need one task per production namespace.

Overview of Data Types

The TypesToPurge parameter in purge tasks (both manual and automated) controls what categories of data are included in the purge operation. InterSystems HealthConnect stores various types of production-related data beyond just messages, including event logs, business rule execution logs, I/O logs, and managed alerts. The TypesToPurge setting determines which of these data types are purged.

All Types Option

"All Types" is the most comprehensive setting, purging all categories of production data that meets the age criteria. This includes message sessions (the primary message flow data), event log entries (detailed logging of production activities), business rule logs (execution history of routing rules), I/O logs (low-level input/output records), managed alerts (historical alert records), and other production-related data. All Types provides the most complete cleanup and frees maximum database space.

Messages Option

"Messages" is a more selective setting that purges only message session data - the messages themselves, their headers, bodies, and directly related metadata. This setting preserves other production data like event logs and business rule logs while cleaning up the primary message data. Messages is appropriate when you want to remove old message content for space management while retaining operational logs for troubleshooting or auditing purposes.

Other Selective Options

Other specific TypesToPurge options allow even more granular control. You can purge only event logs, or only business rule logs, or only I/O logs, while preserving messages. These selective options are less commonly used but can be valuable in specific scenarios - for example, purging verbose event logs that are consuming space while retaining message history for compliance.

Exam Importance

For the certification exam, it's important to know that "All Types" and "Messages" are the most common TypesToPurge values. Exam questions about purge tasks often specify which types should be purged, and you need to recognize the correct setting. A question might show a purge task configured with TypesToPurge = "Events" and ask why messages aren't being purged - the answer is that Events only purges event logs, not messages.

Importance of Verification

Even after configuring automated purge tasks, it's essential to verify they're running correctly and actually purging messages as intended. Several configuration errors can cause purge tasks to fail silently or purge the wrong data, and understanding how to verify correct task configuration is an important skill tested on the certification exam.

Namespace Verification

The most critical check is verifying the namespace setting. Each purge task must be configured to run in the correct namespace where the production exists. A common exam scenario shows a purge task running in the ENSEMBLE namespace when the production is actually in the HEALTHCONNECT namespace. The task runs without errors but doesn't purge any messages because it's looking in the wrong namespace. When troubleshooting purge problems, always verify the task's namespace matches the production's namespace.

Class Name Verification

The task class name must be exactly correct: Ens.Util.Task.Purge. Exam questions sometimes show tasks configured with incorrect class names like Ens.Task.Purge, Ens.Util.Purge, or InterSystems.Task.Purge. These incorrect class names cause the task to fail to run at all. When reviewing purge task configuration, verify the exact class name including correct capitalization and punctuation.

TypesToPurge Verification

The TypesToPurge parameter must match what you actually want to purge. If the requirement is to purge messages and the task is configured with TypesToPurge = "Events", messages won't be purged even though the task runs successfully. Similarly, verify that retention periods are set correctly - a task configured to purge messages older than 365 days won't help if the database is full and you need to purge messages older than 90 days.

Task Status Verification

Check that the task is not suspended. Tasks can be suspended manually by administrators or automatically if they fail repeatedly. A suspended task won't run according to its schedule. The task manager interface shows task status, and suspended tasks are clearly marked. If a purge task isn't running, verify it's in active status rather than suspended.

Schedule Verification

Finally, verify the task schedule and next run date. The schedule should be appropriate for the environment - typically daily for production environments. The next run date should be reasonable - if today is January 15 and the next run is scheduled for February 28, something is wrong with the schedule configuration. Check that the schedule is configured as intended and that the next execution will occur when expected.

Accessing Execution History

The Task Manager provides execution history for all scheduled tasks, including purge tasks. This history is invaluable for verifying that purge tasks are running as intended and for troubleshooting when purge operations fail or don't achieve expected results. Understanding how to access and interpret task execution history is important for production support and exam scenarios.

Interpreting Success and Failure

Task execution history shows each time the task ran, including the start time, end time, and completion status (completed successfully, failed, still running). For purge tasks, successful completion indicates that the purge operation finished without errors, but doesn't necessarily mean messages were purged - if no messages met the purge criteria (all messages are within the retention period), the task completes successfully but purges nothing.

Diagnosing Failures

When a purge task fails, the execution history includes error messages or stack traces indicating what went wrong. Common failure causes include incorrect namespace configuration (task can't find the production), incorrect class name (task class doesn't exist or can't be instantiated), database lock conflicts (purge can't acquire necessary locks to delete records), or resource constraints (insufficient memory or database space to complete the purge).

Performance Monitoring

The execution history can also show performance trends. If purge tasks that normally complete in 10 minutes suddenly start taking hours, this might indicate database fragmentation, excessive message volume, or other performance issues requiring attention. Monitoring purge task duration over time helps identify these trends before they become critical problems.

Exam Relevance

For the certification exam, understand that task execution history is the primary tool for verifying purge tasks are working correctly. If an exam question asks how to determine whether a purge task has been running, the answer involves checking task execution history. If a question asks how to troubleshoot a failing purge task, reviewing the execution history for error messages is the correct first step.