The lease ended before the process stopped
A background worker can continue running after its lease expires. The scheduler has permission to assign the job again, but expiry does not stop the first process or withdraw a request already traveling to storage. If correctness depends on excluding that old worker, the protected resource needs to reject obsolete ownership tokens.
The reconstruction here follows a single report-generation job. Worker A reads an input snapshot, pauses and loses its lease. Worker B takes over and saves a newer result. A then resumes and tries to replace B's result. This is a local deterministic model, not a customer incident. Its event labels and ordering are deliberately explicit so the diagnosis does not depend on a plausible story about a production system we have not inspected.
Start with the write that changed the resource, then trace backward to the authority it carried. A dashboard showing one current lease holder can be completely accurate while a previous holder is still attempting side effects.
Reconstruct ownership at the resource
The model grants A token 41 and B token 42. These are illustrative monotonically increasing ownership values. The protected resource remembers the highest token it has accepted. The model rejects a write with a lower token. Its token check and value change form one indivisible operation.
| Step | Event | Resource consequence with fencing |
|---|---|---|
| 1 | A receives token 41 and reads its input | No result has been written |
| 2 | A pauses; its lease expires | Expiry alone does not modify the resource |
| 3 | B receives token 42 and writes a new result | Resource accepts 42 and records it |
| 4 | A resumes and submits its old result with 41 | Resource rejects 41 because 42 was already accepted |
| 5 | B submits the same result again with 42 | Token check accepts it; duplicate suppression is a separate concern |
Without fencing, the fourth event replaces the new result with the old one. With fencing, the final value remains B's result. The fifth event is intentional: a fencing token orders different owners, but the same owner may make several requests. A policy that accepts equal tokens cannot by itself distinguish a valid second operation from a duplicate of the first. Martin Kleppmann's analysis of distributed locking explains why a pause between an ownership check and a write defeats a client-only check. Hazelcast's fencing documentation likewise describes passing monotonically increasing ownership values to an external resource. The local model below implements only that resource-side comparison. It does not implement either a distributed lock service or a consensus algorithm.
Test the hypothesis that distinguishes this failure
Collect the job ID, worker attempt ID, lease generation and the token observed by the resource. Keep the resource's accepted token with its mutation record. Application logs saying "lock acquired" cannot show whether the eventual side effect used the same ownership generation or whether storage accepted a newer generation first.
A useful reproduction pauses A after it prepares a write, allows its lease to expire and lets B complete a write before resuming A. Pausing A before it acquires the lease tests something else. Killing A permanently also misses the failure: the stale write requires an old actor to return.
The replay program uses only Python's standard library. Save it as replay.py in an empty working directory and run it with python3 replay.py. It writes the event trace and the comparison data. There are no sleeps or real lease timers. Event order is the experimental input, so repeated runs have the same result.
unfenced final value: old-result
fenced final value: new-result
repeated current token: accepted
old token before newer token reaches resource: acceptedThe last line is a critical limit.
A resource that checks only its highest observed token does not learn immediately that a lease expired elsewhere. An old token can still be accepted before a newer token reaches that resource. Fencing establishes an ordering at the resource. It does not revoke a worker at the instant the scheduler's lease expires.
If the requirement is that no old worker may write after the exact reassignment instant, the design needs an authoritative ownership check coordinated with the mutation, not merely a highest-seen-token cache. State the stronger requirement before deciding whether the ordinary fencing protocol is sufficient.
Locate the guard beside the side effect
The token comparison must happen where it can prevent the mutation. A worker can include a token in every log line and still corrupt data if the receiving store ignores it. Similarly, reading last_token, deciding the write is allowed and then updating in a separate unprotected operation recreates a race inside the receiver.
For a database-backed result record, a conditional mutation or transaction can combine the comparison and write. The exact statement depends on the database's concurrency semantics and on whether multiple records belong to one protected resource. The replay uses a single in-memory operation to express that requirement; it is not a substitute for testing the chosen database under concurrent access.
For an external email or payment service that does not accept ownership tokens, adding a fencing field to your own job table cannot retract an already sent request. Identify the provider's supported duplicate-control mechanism and reconcile uncertain outcomes. The API idempotency article covers a different question: how repeated requests for the same logical operation converge on one effect.
Resource identity matters too. If tokens are allocated per job but several jobs mutate the same report, a token from one job cannot automatically be compared with a token from another. The monotonic sequence and the resource's comparison domain must describe the same ownership scope.
Review tempting fixes against the same timeline
| Proposed fix | What it can improve | What the replay still exposes |
|---|---|---|
| Increase the lease duration | Reduce reassignment during ordinary pauses | A longer pause can still outlast the lease |
| Renew the lease more often | Detect some lost workers sooner | A paused process cannot renew or observe its own loss |
| Check the lease immediately before writing | Catch an already visible expiry | A pause or delayed request can occur after the check |
| Reject old tokens at the resource | Prevent older owners from overwriting an accepted newer generation | Equal-token duplicates and not-yet-observed reassignment need separate handling |
The first two changes may be useful operational tuning. Their value should be measured through processing latency and unnecessary reassignment, without describing them as proof of mutual exclusion. Fencing addresses a different failure boundary.
Keep rejected stale writes observable. A rising rejection count may indicate long pauses, delayed requests or work that routinely exceeds its lease. Do not automatically classify every rejected write as a harmful incident: in the replay, rejection is the expected protective behavior. Pair the event with whether the current attempt completed and whether the customer received the right result.
Close the investigation with a surviving invariant
The invariant demonstrated here is narrow: after this resource accepts token 42, a later write carrying 41 cannot replace its value. The replay also records the behaviors that invariant does not cover. That combination is more useful than reporting that a lock test "passed" without saying what was excluded.
Before adopting the pattern, locate the monotonic token issuer, verify that its sequence survives the failures in your design and identify the atomic resource-side enforcement point. Then repeat the paused-worker test against the real receiver. If the receiver cannot reject an obsolete owner, the job's correctness still depends on behavior outside the lease service.
Sources
Documentation checked .