The deployment succeeded, but the same message failed again
A dead-letter queue is shrinking after a redrive, then starts filling with familiar event IDs. The deployment may have fixed one consumer defect while leaving another failure class unchanged. Moving a message back into the processing path does not change the reason it was rejected.
The executed local fixture places four fictional events through two consumer versions. The unchanged version processes two events and returns two to the dead-letter outcome after nine attempts. A version that supports the new schema processes three and returns one after seven attempts. The malformed event still fails under both versions.
This is a constructed investigation, not a report of a production incident. The script uses assigned handler outcomes and a maximum of three attempts per event in each run. It does not run Amazon SQS, model visibility timeouts or test an actual redrive task. The attempt trace makes the causal difference inspectable.
Classify the failure before replaying the backlog
The four events exercise different explanations. ok-1 is already valid. transient-1 fails once and succeeds on its second attempt under the fixture's assigned recovery rule. schema-1 requires a consumer that understands its payload version. poison-1 remains malformed, so repeating it against either consumer changes nothing.
A real queue backlog may mix these categories with expired business requests and events whose side effects already happened. A deployment version is therefore insufficient as a redrive acceptance criterion. Associate a sample with its failure reason, payload contract and processing history before choosing what to replay.
| Event class | Unchanged consumer | Schema-capable consumer | What changed |
|---|---|---|---|
| Valid | Processed on attempt 1 | Processed on attempt 1 | Nothing required |
| Assigned transient failure | Processed on attempt 2 | Processed on attempt 2 | Dependency outcome recovers in fixture |
| New schema | Returns to DLQ after 3 attempts | Processed on attempt 1 | Consumer understands that contract |
| Malformed payload | Returns to DLQ after 3 attempts | Returns to DLQ after 3 attempts | No relevant repair |
Amazon SQS's dead-letter queue guidance describes isolating messages after the configured receive threshold so their failures can be examined. Our three-attempt loop is an educational policy, not a reproduction of SQS receive-count and visibility semantics. Check the actual queue configuration before interpreting operational counts.
If schema handling caused the failure, preserve a versioned payload example in the consumer's acceptance tests. The event-schema evolution article examines compatibility at that boundary. If a payload is malformed rather than merely newer, adding support for a valid schema does not repair it.
Prove what the fix changes with the same inputs
Both runs in the fixture receive the same four events. Only the schema-capable consumer behavior changes. That controlled comparison is why the difference is attributable to the fix: schema-1 moves from repeated rejection to processing, while the other event classes retain their assigned behavior.
The total number of attempts falls from nine to seven. This is not a throughput or cost benchmark. It follows from replacing three failed schema attempts with one successful attempt in this exact sample. The malformed event still consumes three attempts and returns to the dead-letter outcome.
The results file records each event's terminal outcome, not just the aggregate success count. Looking only at the three successful events after the fix would hide the unchanged failure. A useful acceptance report names the cohort that remains unsafe to replay and the evidence required to change that decision.
For a real consumer, use sanitized fixtures that preserve the relevant structure without copying sensitive customer data into a public test package. Run them against the version and configuration intended for the redrive. A locally fixed parser may still behave differently behind a production feature flag or dependency setting.
Redrive identity is not business-event identity
A replay can be a new transport delivery of an old business event. Amazon SQS's redrive documentation states that redriven messages receive new message IDs and enqueue times. A consumer that treats the transport message ID as the identity of the business effect can therefore misclassify a replay as new work.
Keep a stable domain event or operation identifier in the application contract, with durable processing evidence where the effect is applied. If the original attempt charged, sent or changed something before it failed to acknowledge, a replay needs to resolve that state. Authentication or successful parsing alone cannot prove the effect remains unapplied.
The fixture does not exercise this storage boundary. Its event names merely let the reader compare outcomes. The API idempotency cases provide a separate example of stable request identity after uncertain responses. A queue consumer needs an equivalent contract suited to its transaction and external-effect boundaries.
SQS redrive tasks also do not filter or modify messages as part of the task. If the recovery plan depends on selecting a safe subset or repairing payloads, design an explicit controlled workflow with appropriate evidence. Do not assume the redrive control itself performs the classification shown in our worksheet.
Release a reviewed cohort, then watch its actual outcome
A proposed redrive decision should name the consumer version, failure class and stable business identities covered by the evidence. The redrive review sheet leaves decisions pending for new-schema, malformed, transient and already-applied cohorts. It is not an automatic recommendation to replay the entire queue.
Choose a bounded initial rate appropriate to the receiving system, then compare processed outcomes, repeated failures and duplicate-effect evidence. AWS exposes redrive velocity controls, but a configured speed is not proof that downstream systems can absorb it. The acceptance signal is the observed result at the consumer and business boundary.
Ordering also needs an explicit decision. Moving failed messages aside can allow later work to proceed, and redrive can interleave older business events with current traffic. AWS cautions that using a DLQ can conflict with strict FIFO ordering requirements. Validate the application's state-transition policy before reintroducing delayed events.
If the same event returns, preserve its failure evidence and stop treating another identical replay as a new experiment. Either the relevant input, consumer behavior or dependency condition must change, or the event needs an explicit disposition. In the executed fixture, the schema fix earns one additional success. It supplies no evidence that the malformed cohort is ready.
Sources
Documentation checked .
Dreamtsoft Editorial
The unchanged malformed event is the strongest reason to inspect individual outcomes. A better aggregate success count does not make that event ready for another replay.
Dreamtsoft Editorial
Keeping the input cohort fixed makes the schema fix reviewable. Replacing the failing payload with an easier example would lose the evidence for what the consumer change repaired.
Dreamtsoft Editorial
A replay review should ask whether the original business effect already happened before acknowledgement failed. Parsing the message successfully does not answer that question.
Dreamtsoft Editorial
The distinction between transport identity and business identity belongs in the acceptance sheet. A new delivery identifier should not authorize repeating an old business operation.
Dreamtsoft Editorial
Which failure signal would stop the first redrive cohort? A queue that drains while downstream operations fail would be a poor completion signal.
Dreamtsoft Editorial
Delayed events need a state-transition review before they mix with current traffic. Their original order may matter even when the receiving consumer can parse every payload.