Six stops reveal three different failures
An order can exist in the database while its notification never reaches the next service. A transactional outbox addresses that gap by committing the business change and an event record together. A separate relay publishes committed events. The relay still has a failure boundary of its own.
We executed six controlled stop/restart schedules using SQLite and a small in-memory broker model. Each case begins with one proposed order. The table distinguishes committed orders, pending events at restart, accepted deliveries and consumer effects. Those counts answer different questions.
| Controlled case | Committed orders | Pending at restart | Total deliveries | Consumer effects |
|---|---|---|---|---|
| Direct-write gap | 1 | 0 | 0 | 0 |
| Stop before commit | 0 | 0 | 0 | 0 |
| Stop before relay | 1 | 1 | 1 | 1 |
| Publish then stop | 1 | 1 | 2 | 1 |
| Mark then stop | 1 | 0 | 0 | 0 |
| No consumer dedup | 1 | 1 | 2 | 2 |
The direct-write case commits the order and stops before sending its event. Nothing in that design records the remaining obligation. The atomic transaction's pre-commit stop leaves neither an order nor an event. After an atomic commit, a pending event survives the modeled relay restart and can be sent.
Two deliveries do not necessarily create two effects. The fourth case records a consumer receipt with its local effect, so the repeat is ignored. The last case removes that protection and applies the effect twice. These are deterministic model outcomes, not broker performance measurements or tests of operating-system crash durability.
Put the event inside the business transaction
The useful local boundary contains the order write and the outbox insertion. Both operations must use the same database transaction. If the transaction fails, neither change should become committed state. If it succeeds, the database contains an explicit obligation that another process can inspect.
AWS describes this pattern as a response to the dual-write problem and shows an outbox updated in the same transaction as business data. Its guidance also identifies duplicate delivery and message ordering as separate concerns. AWS transactional outbox guidance.
In our SQLite example, the producer starts a transaction, inserts an order and inserts an event. A controlled exception before commit triggers an explicit rollback. The relay queries only after this producer phase is complete. That schedule keeps the local transaction question separate from concurrent-worker behavior, which the model does not exercise.
An event should carry enough identity to explain what happened. A practical design may include an event ID, tenant ID, aggregate ID, event type and payload version. Decide which values describe the committed change and which can be looked up later. Looking up the latest order state during delayed delivery can change the meaning of an event that originally described an earlier transition.
The sequence is: commit business data with an event; find pending committed events; publish; record completion; apply the consumer effect with its receipt. The producer transaction ends before publication. The consumer transaction belongs to the receiving service. The diagram does not imply one transaction covering every box.
Record completion after publication
The relay's hardest moment comes after the broker accepts a message but before the relay records that acceptance. If the relay stops there, the outbox still looks pending. A restart can publish the same event again.
That behavior is explicit in the transactional-outbox pattern description: a relay can publish more than once when it stops between sending and recording completion. The suggested response is an idempotent consumer that tracks processed message IDs. Transactional outbox pattern.
Moving the completion record before publication creates a different problem. Our fifth case sets the sent flag, commits it and stops before adding the event to the broker model. The restart scan finds no pending row. The order remains committed, but there are no deliveries and no consumer effects.
This is why a sent flag needs a precise meaning. It should represent a defined acceptance condition, rather than an intention to attempt delivery. In a real integration, specify which broker acknowledgement is sufficient and what happens when the acknowledgement is missing or ambiguous. The in-memory list in this example represents acceptance immediately; it cannot validate a real broker's persistence guarantees.
Make the consumer receipt atomic with its effect
A receipt table is useful only if its relationship with the business effect is correct. Recording a receipt and then failing before applying the effect can suppress necessary work. Applying the effect and then failing before recording the receipt can repeat the work.
The model uses a separate consumer database. For each delivered event, it attempts to insert a receipt with a unique event ID. A new receipt and the effect are committed in one local transaction. If the receipt already exists, the consumer skips the effect. The two deliveries in the fourth case therefore produce one stored effect.
This guarantee is limited to that local database. Sending an email, charging a payment method or updating another service introduces an additional boundary. An external action needs an appropriate idempotency contract or another durable handoff. Our API idempotency cases examine how a repeated request relates to a previously committed result.
Receipt retention is also part of the contract. If the system removes receipts before old events can be replayed, a later duplicate may look new. Define the replay horizon and retention policy together. Do not infer a safe retention duration from the tiny fixture used here.
Preserve identity and define ordering scope
Duplicate detection depends on stable identity. A retry of the same event should retain its event ID. A genuinely new business transition should receive a new identity even when its payload resembles an earlier one. Payload equality alone cannot establish whether two messages represent one obligation.
Ordering requires a separate decision. Many products need order within one account or aggregate, while unrelated tenants can proceed independently. State that scope before choosing a partition key or worker assignment. A global ordering requirement can impose coordination that the product does not need.
The example has one event and one relay, so it provides no evidence about ordering across multiple events. A production test should include two transitions for the same aggregate, independent transitions for another tenant and a delayed retry of an earlier event. Our webhook delivery examples show why duplicate handling and out-of-order handling need different checks.
Parallel relays also need an ownership rule. A row claim, lease or database locking strategy can coordinate workers, but each introduces expiry and retry cases. Do not copy the sequential SELECT loop from the download into a multi-worker service and assume it supplies that coordination.
Observe obligations that are still unfinished
A low queue depth does not prove successful delivery. In the mark-before-publish case, the final pending count is zero even though the event never reaches the consumer. Metrics should be tied to the meanings of state transitions and reconciled with business expectations.
Useful operational questions include how old the oldest pending event is, whether attempts keep failing for the same event and whether a tenant's work has stopped progressing. Separate an expected duplicate from an unexpected repeated business effect. Otherwise an alert can treat normal retries as failures while overlooking lost obligations.
A poison event needs an explicit handling path. Decide how operators inspect its failure, correct the underlying problem and replay it without changing its identity accidentally. A quarantine state should preserve the obligation and its reason. Silently dropping the row merely makes the pending metric look better.
Schema changes deserve the same attention. Old pending payloads may remain after a new producer or consumer is deployed. Define the accepted payload versions and test their interpretation during a rolling deployment. A successful database migration does not automatically make an old event understandable to a new consumer.
Run and adapt the model
Download the Python source, result table and complete traces. Save the source as experiment.py and run it with Python 3. It uses only the standard library and creates its databases in memory.
python3 experiment.pyThe script executes the schedules twice and asserts identical outcomes. It also checks the four main counts in every row. The direct-write gap and premature sent marker are deliberately broken alternatives, included to make their consequences visible.
A controlled stop is represented by a Python exception, not a killed process or machine failure. The broker is a list retained across relay calls. There is no network, concurrent relay, distributed transaction, acknowledgement loss or external consumer action. Those boundaries are named so the example can guide a fuller test suite without claiming to replace one.
When adapting the model, change one boundary at a time. First test producer rollback. Then test relay restart after acceptance. Finally test consumer restart around its receipt and effect. Keep event IDs and resulting business state in the assertions; a test that checks only whether a handler returned successfully can miss the failure you intended to study.
Review the delivery contract before release
| Boundary | Evidence to request | Failure that must remain visible |
|---|---|---|
| Producer transaction | Business row and outbox row commit or roll back together | Committed business change without an event obligation |
| Relay completion | A defined acceptance signal precedes the sent marker | Accepted message whose local completion is unknown |
| Consumer transaction | Receipt and local effect commit together | Receipt without effect or effect without receipt |
| Replay and retention | Stable event identity and documented retention horizon | Old delivery becoming indistinguishable from new work |
Use this table when reviewing an implementation or an incident. Identify the last durable fact each participant can prove, then determine which action is safe to repeat. That exercise is more informative than describing the whole pipeline with one delivery guarantee.
Sources
Documentation checked .