What Breaks When an HL7 Feed Goes from 500 to 10,000 Messages a Day
By Waseem Ahmad — Full Stack Developer & AI Engineer ·
TL;DR
- A missed HL7 message is not a technical nuisance — it is a patient chart that is wrong, and nobody knows it yet.
- The four failures that appear at volume — message ordering collapse, duplicate ADTs, retry storms, and silent loss — rarely surface in testing.
- Retry logic is the most common source of both duplicate records and silent message loss; the two problems share the same root cause.
- Partitioning ADT events by patient identity, not by message control ID, is the single architectural decision that prevents the most downstream damage.
- The interface that hit 99% delivery across 10,000+ clinical messages a day on the Fukuda Denshi USA gateway got there through explicit dedup keys, dead-letter queues, and queue-depth alerting — not through a bigger server.
Start with what is at stake before getting into protocol mechanics. A patient is admitted to a cardiac monitoring unit. An ADT A01 admission event fires. The HL7 feed is running fine — five hundred messages a day, one monitor per channel, low traffic. Then the unit expands. Volume triples. Then doubles again. The feed is still technically running. But one admission event arrives at the EMR out of order, behind a transfer event that had already moved the encounter. The chart shows the patient as transferred before they were admitted. A nurse reconciling vitals sees the conflict, pauses, calls the floor. The fifteen minutes that follow are recoverable. On a busier unit, with less attentive staff, they are not.
That is the cost the architecture is responsible for. Not the error log. Not the queue depth metric. The chart.
I ran the HL7 IoT gateway for Fukuda Denshi USA — 200+ patient monitors, 10,000+ clinical messages a day, with a target of 99% delivery. What follows is what I learned about the four failure modes that do not show up at 500 messages a day but arrive reliably at 10,000.
Failure Mode 1: Message Ordering Collapses
ADT traffic looks simple until patient identity and encounter state start changing at real-world speed. At low volume, events arrive close enough to in-order that the engine's default processing handles them without incident. At scale, the gap between "received" and "processed" widens, and two messages for the same patient can be picked up by different worker threads and land out of sequence.
The architectural trap here is using the wrong partition key. ADT events must be keyed by a composite of patient MRN plus facility OID so that every event for one patient lands on one partition and is processed sequentially. MSH-10 (Message Control ID) is the correct key for deduplication, but using it as a partition key scatters one patient's events across partitions and destroys ordering — the single most common cause of orphaned encounters in production.
The A08 and A40 event types cause a disproportionate share of production incidents precisely because teams test them last, if at all. An A08 (update patient information) that arrives before the A01 it references creates an encounter that downstream systems cannot resolve. An A40 merge that arrives out of sequence can corrupt the Master Patient Index in ways that take days to untangle. Out-of-order ADT A40 merge messages can split or cross-link medical records, corrupting historical patient identities.
Failure Mode 2: Duplicate ADTs Accumulate Silently
Duplicate ADT messages are endemic — network timeouts, vendor retransmission policies, and manual resends all produce them. At 500 messages a day, a duplicate here and there gets caught by staff during routine chart review. At 10,000 messages a day, it becomes a background rate of corruption that nobody catches until someone runs a reconciliation query and finds two active encounters for the same patient visit.
The mechanism is straightforward. A network blip prevents an ACK from reaching the sender, triggering a retry. The original message was delivered and committed. The retry delivers it again. If the receiver is not idempotent, you now have two records. Blindly resending can create a duplicate if the receiving application already committed the original transaction.
The fix requires two things working together: MSH-10 as a dedup key checked before any write, and ACK semantics that are honest. That starts with honest acknowledgments — don't return AA if you couldn't process the message — and continues with persistent storage of every message and its outcome. An AA sent before the downstream write is the most common source of duplicates in feeds I have reviewed. The sending system believes delivery was confirmed. The write fails. The sender never retries. The message is gone.
Is Your Interface Engine Actually Designed for This Volume?
This is the question worth asking before anything else. Most interface engines are configured for the volume they were deployed at, not for the volume they will reach. Traditional engines cannot dynamically add workers during peak shifts or scale down overnight; scaling requires manual provisioning, load-balancer changes, and additional licenses. The configuration that worked at 500 messages a day — thread pool size, queue depth, MLLP connection timeouts — may be entirely wrong at 10,000.
| Failure Mode | Trigger at Volume | Clinical Consequence | Primary Fix |
|---|---|---|---|
| Message ordering collapse | Multiple worker threads, wrong partition key | Orphaned encounters, wrong patient state in downstream systems | Partition by MRN + facility OID, not MSH-10 |
| Duplicate ADTs | ACK timeout triggers retry after successful write | Two active encounters for one visit, chart conflicts | MSH-10 dedup key at receiver; honest AA semantics |
| Retry storm | Downstream slowdown + aggressive retry interval | CPU exhaustion, MLLP drops, cascading outage | Exponential backoff, capped retry counts, DLQ |
| Silent message loss | MLLP framing error, dead MLLP socket, full DLQ | Missing results, vitals, or admission events with no alert | Queue-depth alerting, TCP keepalive, MSH-10 sequence gap checks |
Failure Mode 3: Retry Storms
Retry storms are where volume turns a recoverable partial failure into a full outage. The sequence is consistent. A downstream system slows under load — database saturation, a slow ORU batch, a network hiccup. MLLP ACKs start arriving late. The sender, seeing no ACK within its configured timeout, begins retrying. If downstream database load caused the timeout, duplicate messages every ten seconds can worsen CPU exhaustion and cause further MLLP connection drops. The retry pressure makes the downstream system slower, which causes more timeouts, which causes more retries.
Two configuration mistakes amplify this. First, a retry interval that is fixed rather than exponential. Retry transient failures with exponential backoff so you don't hammer a struggling endpoint, and cap retries or retry duration — infinite retries on a permanent error hide the problem. Second, no differentiation between a transient connectivity failure and a permanent data error. Connectivity loss, invalid segment content, downstream application rejection, duplicate detection, and business-rule violations should not all follow the same path. A malformed message that can never succeed should go to a dead-letter queue immediately; retrying it forever consumes worker threads that legitimate messages need.
Failures here stem from parsing errors on non-standard delimiters, unhandled custom Z-segments, script exceptions or thread pool exhaustion during volume spikes. A lab ORU batch that arrives at shift change can consume all available threads, and the ADT admission messages that follow sit in the inbound queue until a thread frees. Using separate protocol-specific inbound channels, dedicated transformation channels, and outbound channels with retry logic — rather than putting receive, transform, and send logic in one monolithic channel — is the structural change that prevents one message type from starving another.
Failure Mode 4: Silent Message Loss
Silent loss is the worst failure mode because nothing alarms. Silent loss is the failure mode that erodes clinical trust, because it surfaces as 'the result never arrived' with no evidence of what happened. By the time a clinician notices a missing result or a missing admission event, the message is gone and the sending system has moved on.
There are several distinct mechanisms. MLLP framing errors are durable precisely because they are invisible to most monitoring. A message that fails on framing usually produces no application-level error, no validation failure, and no ACK — just silence or a generic socket timeout. A second mechanism is the dead MLLP socket: dead MLLP TCP/IP sockets, expired TLS/SSL certificates, dropped IPsec VPN tunnels, or firewall changes silently blocking traffic between facilities all produce the same symptom — flatlined ingestion metrics while the source system believes it is connected. A normally busy ADT interface that suddenly receives nothing may deserve attention even when no explicit error appears.
A third mechanism is a dead-letter queue that is full or unmonitored. Dead-letter queues can silently lose failed messages when full or misconfigured, especially without regular monitoring. The DLQ is supposed to be the safety net. When nobody is watching it, it becomes a second place where messages disappear.
Detection requires monitoring that goes beyond CPU and connection state. Check whether MSH-10 Control IDs are sequential; gaps indicate lost messages. Queue-depth alerting — an alert that fires when a normally busy inbound channel receives nothing for five minutes during business hours — catches silent failures that no error log will ever surface.
What the Fukuda Denshi Gateway Actually Used
The HL7 IoT gateway I built for Fukuda Denshi USA handles 10,000+ clinical messages a day from 200+ patient monitors at 99% delivery. The architecture decisions that got it there were not exotic. They were the ones described above, applied consistently.
Dedup keys on MSH-10 with a short-TTL Redis cache before any downstream write. ADT events partitioned by device-and-patient context so that events for the same source arrive sequentially. Exponential backoff on retries with a hard cap, and a dead-letter queue that alerts within minutes of the first message landing in it. Per-message-type inbound channels so that a burst of waveform data cannot delay an alarm event. And TCP keepalive on every MLLP connection, because a silently dead socket at 3 a.m. on a 200-monitor deployment is a patient safety issue, not an inconvenience.
The result was 95% of manual data entry eliminated and an estimated $400K/year in operational savings — not because the engine was fast, but because it was observable and the failure modes were enumerated before they happened in production. You can see the case study at /work/mirth-connect-dashboard.
If you are evaluating an existing integration before going live, the checklist in Healthcare Integration Due Diligence: Questions to Ask Before Sign-Off covers the exact verification steps — dedup testing, retry simulation, dead socket detection — that catch these failure modes before they cost you a chart. And if you are weighing what a HIPAA audit will actually examine in the integration layer, see What a HIPAA Audit Actually Checks in Your Integration Layer.
If your feed is approaching the volume where these failure modes become likely, or you are already seeing unexplained duplicates or missing messages, the right next step is a structured interface review. That is the kind of engagement I do through healthcare software services — audit the configuration, instrument the failure paths, and fix what will break before it breaks on a patient chart.
FAQ
How do I know if my HL7 interface is silently dropping messages?
The most reliable signal is MSH-10 sequence gap monitoring: if the Control IDs are sequential and a gap appears, a message was lost. Pair that with queue-depth alerting — if a normally busy inbound channel goes quiet during business hours, that is a silent failure until proven otherwise. Application-level monitoring ("we received N admission messages in the last hour, but the EMR shows N-3 new encounters") catches the gaps that protocol-level monitoring misses.
Why do retry storms get worse the more you retry?
When ACKs fail to return, systems may freeze the active socket or enter retry loops, repeatedly resending messages and consuming network bandwidth. Each retry adds load to a downstream system that is already struggling. Without exponential backoff, the retry interval stays constant, the pressure stays constant, and the downstream system never gets the quiet period it needs to recover. A fixed-interval retry on a saturated endpoint is effectively a denial-of-service from inside your own network.
Is this a Mirth Connect problem specifically, or does it affect all interface engines?
These failure modes are not engine-specific — they are properties of the MLLP transport and HL7 v2 ACK semantics, which all major engines share. Mirth Connect, Rhapsody, Corepoint, and Qvera all expose the same retry, ordering, and dedup surface area. The difference is in how each engine's configuration exposes those controls, and how much operational discipline the team running it applies. Engine choice matters less than whether retry logic, partition strategy, and DLQ alerting are configured explicitly.
When does a feed NOT need this level of hardening?
If your interface is unidirectional, low-stakes (non-clinical administrative data), and runs under a few hundred messages a day with human review on the downstream side, a simple Mirth channel with basic retry and email-on-error is probably sufficient. The hardening described here pays for itself when message volume is high enough that manual review cannot catch failures, when the downstream data affects clinical decisions, or when a duplicate or missing record cannot be remediated by simply asking the source to resend. If none of those apply, the simpler configuration is the right one.