The record disappeared, then returned
A search result that remains visible immediately after deletion may reflect refresh timing. A result that disappears and later returns needs a different investigation: something may have written an older document back into the projection. Capture both the source revision and the projection's last accepted event before changing refresh settings.
The example here is a constructed, executed Python fixture. It is not a report of a customer incident or a test of an Elasticsearch cluster. A source record receives revision 10, is deleted at revision 11, and then an old revision-10 upsert arrives late. The source remains deleted throughout that late delivery. Only the search projection changes.
One implementation physically removes the projected row and its remembered revision. Another keeps a small deletion marker containing revision 11. Both appear correct just after the delete. Their behavior diverges on the next delivery, which is why checking only the immediate response misses this failure.
Download the executed model, complete results and 15-row delivery trace. Save the model as experiment.py and run python3 experiment.py in an empty directory. It creates its own local result files and uses no external service.
Three explanations require different evidence
Start with the result's identity, including tenant, source ID, index and routing identity where applicable. A deletion against the wrong identity can resemble replay while never removing the intended document. Then compare source state, accepted indexing operations and what the query actually returned.
| Candidate explanation | Evidence that supports it | Check that separates it |
|---|---|---|
| Search visibility delay | Delete accepted; no later write for that identity | Observe search after the relevant refresh and inspect write history |
| Stale event accepted | Older source revision was indexed after the delete | Compare source revisions in accepted operations, not just arrival timestamps |
| Identity or query mismatch | Delete and hit belong to different routing, tenant or index scope | Match the complete identity and query target |
Elasticsearch's delete API documentation distinguishes refresh behavior, routing and the temporary retention of a deleted document's version. These are separate mechanisms. A successful delete does not by itself establish that every future indexing request will carry a valid source revision or that an arbitrarily old replay will be rejected.
An application log saying delete sent is weak evidence for all three hypotheses. Capture the destination response and the event's source version. If the consumer retries a batch, inspect individual item outcomes as well as the surrounding request result. The useful chronology is the chronology of accepted mutations for one identity.
Follow the revision that was forgotten
In the physical-delete model, revision 10 creates the row. Revision 11 removes it. When revision 10 arrives again, the consumer sees no stored row and therefore no higher revision to compare. It accepts the old payload as though the identity had never existed.
The retained-marker model also removes the document from visible results, but keeps revision 11 in its projection state. The older event fails the strictly-greater comparison. A duplicate revision-11 delete fails the same check and leaves the marker unchanged.
| Delivery | Physical deletion | Retained deletion marker |
|---|---|---|
| Upsert at revision 10 | Visible at 10 | Visible at 10 |
| Delete at revision 11 | No row or remembered revision | Hidden at 11 |
| Late upsert at revision 10 | Visible again at 10 | Rejected; still hidden at 11 |
| Duplicate delete at revision 11 | Removes the resurrected row | Rejected; still hidden at 11 |
| Authorized recreation at revision 12 | Visible at 12 | Visible at 12 |
Revision 12 is explicitly an authorized new source mutation in this fixture. It is not assigned by a retry worker to make a conflict go away. If a failed consumer manufactures a fresh version for an old payload, the comparison no longer represents source order.
Elasticsearch 8.19's external versioning contract accepts an incoming external version when it exceeds the stored version, or when no document exists. That is useful for out-of-order source updates. It also explains why the lifecycle of the stored comparison state matters. The Python fixture demonstrates this information boundary; it does not reproduce all Elasticsearch deletion internals.
A marker needs a replay boundary
Keeping a tombstone for an arbitrary period only moves the question. The third fixture run deletes its retained marker just before the late revision-10 upsert. The record returns again. No timing benchmark is involved: the test deliberately removes the guard to isolate its role.
Choose the guard lifetime against the oldest event the system can still accept. Include delayed consumers, retry queues, manual replay, import jobs and rebuild procedures. If an operator can replay years of source history, a short normal-delivery delay is not a sufficient retention argument. Conversely, a proven replay floor may permit smaller retained state, provided every writer enforces that floor.
The comparison and mutation must also be atomic in the real projection. A worker that reads revision 11, pauses and later writes without a conditional guard has not implemented the model's rule. The existing conditional-write example shows why moving a check into the persistence operation matters, although its HTTP client scenario differs from an indexing consumer.
Retaining a minimal identity and revision avoids retaining the old document body merely to reject it. That marker still has a data purpose and lifecycle. This article does not prescribe a universal retention duration or a privacy policy.
Rebuilds must preserve the same promise
A fresh index creates another empty-state boundary. A rebuild that imports only currently live records, then accepts an older stream without a coordinated cutover, can recreate deleted identities absent from that snapshot. Define the snapshot boundary and subsequent event range together. A snapshot timestamp alone is insufficient unless the source's consistency and event-position contract make it authoritative.
Use the investigation worksheet to repeat the sequence through every actual writer. Test the normal consumer, retry path, backfill and fresh-index cutover separately. Include duplicate deletion and legitimate recreation so that a fix does not permanently block a valid later source change.
Acceptance requires more than a clean query immediately after deletion. Preserve evidence that the old event reached the consumer, was rejected for the intended reason, and left the document hidden. Also verify that a permitted newer mutation becomes visible. If protected content is involved, enforce current authorization at the application boundary; a search filter over deletion markers alone is not a complete access-control design.
The decisive observation is small: which durable fact told the late writer that revision 10 was obsolete? When that fact is absent, another refresh cannot supply it.
Sources
Documentation checked .