A unique event can still be stale
A SaaS integration receives a subscription update and changes the customer's seat allowance. Then an older update arrives. Its event identifier is new to the receiver, so a duplicate check accepts it and overwrites the newer state. Every event was processed once, yet the application is wrong.
Our runnable example separates those two questions: have we seen this event, and should its contents replace the state we currently hold? It models three application-owned events with explicit revisions. Processing all six possible arrival orders produces the correct final seat count in two orders with event-ID deduplication and all six with a revision guard.
This result applies to replacement snapshots carrying a trustworthy monotonic revision for one object. It does not imply that every provider supplies such a field. It also does not apply unchanged to a stream of financial postings or other events whose individual effects must all be preserved.
| Policy | Correct final state across six orders | Repeated event IDs skipped? | Stale state blocked? |
|---|---|---|---|
| Apply in arrival order | 2 of 6 | No | No |
| Deduplicate event IDs | 2 of 6 | Yes | No |
| Deduplicate and guard revisions | 6 of 6 | Yes | Yes, under this model's revision contract |
The example uses no network or webhook provider. These are deterministic correctness checks, not a delivery success rate, performance benchmark or measurement of Dreamtsoft infrastructure.
Reproduce the overwrite in five deliveries
The producer starts with revision 1 setting seats to 10, then revision 2 setting seats to 30, then revision 3 setting seats to 20. The intended final value is 20. We deliberately deliver revisions in the order 3, 1, 2, 2, 1 to combine reordering with duplicate delivery.
| Delivery | Event revision | Event seat value | Arrival-order state | Event-ID-only state | Revision-guard state |
|---|---|---|---|---|---|
| 1 | 3 | 20 | 20 | 20 | 20 |
| 2 | 1 | 10 | 10 | 10 | 20 |
| 3 | 2 | 30 | 30 | 30 | 20 |
| 4 | 2 | 30 | 30 | 30 | 20 |
| 5 | 1 | 10 | 10 | 30 | 20 |
Arrival-order handling performs five updates and finishes at 10. Event-ID deduplication performs three updates and finishes at 30. The revision guard applies revision 3 once, records older events as stale and ignores their repeated deliveries. It finishes at 20 after one update.
The three policies receive identical inputs. Their different outcomes come from the admission rule for a state replacement. A delivery identifier prevents repeating one event's processing; an object revision prevents an older snapshot from replacing a newer one. Neither field should be inferred from the other.
Separate receipt from completion
A webhook endpoint usually has less work to do than the background job it triggers. Its first responsibility is to authenticate the delivery and establish durable ownership of accepted work. Once that handoff is committed, a worker can process the event independently of the HTTP connection.
In this proposed design, the incoming delivery passes through the following boundaries:
- Verify the sender using its documented signing procedure before trusting the payload.
- Resolve the provider account and tenant from an authorized mapping.
- Persist a uniquely identified inbox record in durable storage.
- Acknowledge receipt after that write succeeds, within the provider's response deadline.
- Have a worker claim the record, apply the business rule and record completion.
An inbox can be the queue itself, or a database table consumed by workers. If you also publish a separate queue message, avoid an unprotected dual write between the inbox commit and that message. Polling committed inbox rows or using a transactional outbox is one way to preserve the work handoff. AWS's outbox guidance also points out that dispatch can duplicate messages, so the consumer still needs its own handling.
Do not acknowledge a valid new event merely because it was placed in process memory. A crash after the response can erase the only copy you own. Likewise, do not mark an inbox record completed before the associated local business change commits. The API idempotency retry cases show why the position of that commit changes what a retry can safely do.
Choose a rule for the event's meaning
An explicit revision guard is appropriate when the producer assigns an increasing version within an object's lifetime and each event describes replaceable state. Apply the update only if its revision is newer than the stored revision. In a real database, the comparison and update must be atomic; two workers should not both decide against an outdated read.
When an event contains an additive business effect, dropping it because a newer event arrived can lose data. A ledger entry, inventory movement or email-triggering workflow may require processing every distinct operation with its own idempotency boundary. A current-state projection can discard an old snapshot while another consumer still preserves that event for its separate business purpose.
| Event contract | Appropriate starting point | Question to resolve |
|---|---|---|
| Full state plus monotonic object revision | Conditional replacement by newer revision | Can the object be recreated or its revision reset? |
| Notification that an object changed | Retrieve authoritative state and reconcile | How are racing reads and stale provider responses handled? |
| Independent additive operation | Deduplicate by durable business identity | Which effects must happen for every distinct operation? |
| Ordered deltas with sequence numbers | Track sequence and detect gaps | How will missing predecessors be obtained or buffered? |
An object identifier may need an incarnation or generation identifier if deletion and recreation can reset its version. Define that scope with the producer. Otherwise a high revision from the old object can suppress all updates to the new one.
For notification-style integrations without a trustworthy ordering field, a fetch of current state is often more useful than guessing event order. That fetch needs its own concurrency policy. Two workers can retrieve different snapshots and complete in the opposite order, so replacing event ordering with HTTP calls does not automatically remove the race.
Check the actual provider contract
Stripe documents duplicate deliveries and non-guaranteed event order. Its snapshot event creation timestamps use seconds and can be shared by distinct events. They should not be treated as the monotonic object revisions used in our model. Stripe describes event-ID tracking for duplicates and retrieving missing related objects through its API.
For its request authentication, Stripe requires the unmodified request body for signature verification. The same guidance recommends asynchronous processing and a quick successful response. These requirements concern delivery handling; they do not grant an application-level exactly-once outcome.
GitHub's webhook guidance gives a concrete response deadline of 10 seconds and documents that a requested redelivery retains the original X-GitHub-Delivery value. Provider-specific identifiers and retry behavior belong in the integration contract. Do not assume one provider's defaults apply to another.
Scope your inbox identity to the provider and relevant account or endpoint context where required. Keep a separate representation of business identity when multiple distinct event objects can refer to the same action. A received-event table is useful evidence, but its presence does not prove that all downstream consequences completed.
Make failed work visible and replayable
Track received, claimed, completed, stale and failed records separately. An accepted HTTP response establishes receipt; a completion record establishes that the worker finished the promised local effect. The gap between those states is a queue-age and reliability signal.
Give a failed record a reason and a next action. Transient dependency failures may justify another attempt. An unsupported event schema may require a code change. A missing tenant mapping needs reconciliation before the worker can safely decide where the event belongs. Replaying every failure without classification can turn one bad record into an endless source of load.
Use leases or another explicit ownership mechanism for worker claims, with a recovery path when a worker disappears. Preserve enough delivery evidence to investigate failures without placing secrets or unrestricted payload copies into logs. Authentication failures should never become trusted business events merely because an operator clicked Replay.
Backlogs also compete with interactive work. If one tenant floods a shared processor, tenant-aware queue scheduling can help govern worker admission. That scheduling control does not replace the inbox, identity or state-transition rules described here.
Test the projection and the handoff separately
Download the projection model, save it as experiment.py and run it with Python 3. It writes results.json beside the script. The saved output includes all five delivery steps for each policy, the six arrival-order checks and the input events.
The script also inserts each of the three possible duplicate events into each of four positions in every permutation. All 72 extension checks preserve the guarded final value. This finite test set establishes behavior for these inputs only. It does not test signature verification, persistent inbox storage, simultaneous workers or crash recovery.
Add a separate handoff test in your service: stop execution after the inbox commit, after the business commit and before the completion marker. For each stop, restart the worker and inspect both the durable business state and the inbox record. The acceptable outcome depends on what the event means, but it should never depend on assuming that a webhook will arrive only once or in a convenient order.
Sources
Documentation checked .