Make the replay promise explicit
Keep a new event reader compatible with every message version it may encounter. If a service can replay a year of events, testing it against yesterday's producer leaves most of that promise unexamined. The relevant boundary is the retained data and the readers that may return, including the version available during a rollback.
Start with two inventories. One lists writer versions still present in the replay range. The other lists reader versions allowed to run, including paused consumers and recovery tooling. A deployment calendar cannot replace either list: an old message can outlive the software that created it.
For a team that owns its consumers and needs historical replay, our proposed decision is to preserve backward compatibility across the supported history and upgrade readers before emitting the changed contract. Independent customer consumers may require compatibility in both directions or a separate versioned stream. That choice changes the release boundary, so make it before selecting a registry setting.
Confluent's compatibility documentation distinguishes backward checks, where new readers accept old data, from forward checks, where old readers accept new data. Transitive checks include the full registered history rather than only the preceding version. Those definitions describe a schema relationship. They do not certify business behavior.
A default can hide the break
Consider a constructed account event with an account identifier and a region. Version v1 readers accept a missing region by substituting unknown. Version v2 removes the region field. Version v3 brings it back as a required field. Each writer in our local fixture emits one representative record.
The old reader can process the v2 record because it owns a default. The v3 reader cannot. It requires a region that the v2 message never contained. Reintroducing the field in today's producer does not insert that value into yesterday's messages.
This is a field-projection example, not an Avro compatibility implementation. It checks presence and reader defaults for three explicitly defined record shapes. It does not model types, unions, aliases or wire encoding. The Avro schema-resolution specification supplies the underlying distinction: a reader can supply its default when a field is absent from the writer's record, while an absent required reader field is an error. A writer default is not a substitute for a reader contract.
Run the Python example with the standard library. It produces the nine pair results and a CSV matrix. The local run accepted eight pairs and rejected one. These counts describe this fixture, with one message per writer, and say nothing about production event coverage.
Read the whole compatibility matrix
Rows are readers; columns are messages. Keeping that orientation visible prevents a common review error: approving an update because the old consumer accepts a new record when the actual release needs the new consumer to replay an old one.
| Reader | v1 message | v2 message | v3 message |
|---|---|---|---|
| v1 | Reads west | Defaults to unknown | Reads east |
| v2 | Ignores region | Reads account | Ignores region |
| v3 | Reads west | Rejects missing region | Reads east |
The rejection is useful, but the successful cells deserve review too. The v1 reader accepting v2 does not mean the region remained accurate. It means the program can continue with unknown. If that value determines routing, tax treatment or data placement, acceptance alone is the wrong release gate.
Our example also shows why a passing current-to-current check is weak evidence. Every diagonal cell succeeds because each writer supplies its own expected record. The difficult cells are the ones created by independent deployment and retained history. Add actual payload boundaries there: omitted optional fields, previously unseen enum values and records produced before a corrective release.
Meaning survives outside the schema
A field can remain a number while changing from seconds to milliseconds. Both records may deserialize. Their meaning is different. A syntactic compatibility checker cannot decide which deadline the customer intended from the field type alone.
Record semantic constraints beside the schema: unit, allowed absence, identity scope and the event that the message claims occurred. For an account event, region might mean the customer's declared region, the current compute location or the location when the event happened. Those are different facts even when all three use the same string vocabulary.
Do not silently replace one meaning with another. Introduce a new field or event contract, define its source of truth and make the transition visible to consumers. If historical data cannot supply the new fact, preserve an explicit unknown state or restrict the operation that requires it. A guessed value creates a different failure from a rejected record and is harder to detect later.
The transactional outbox article covers whether a committed change reaches the event pipeline. This decision starts after that boundary: which interpretation a reader may safely assign to the delivered record.
Choose a rollout boundary
Use an additive transition when old and new meanings can coexist. Publish the proposed contract, test supported readers against representative historical records and deploy readers that understand both states. Emit the new field only after the required consumer population is ready. Remove the old path when its replay and rollback obligations have ended, rather than when the producer deployment finishes.
A separate versioned stream becomes reasonable when the new contract changes event identity, ordering assumptions or required facts that cannot be reconstructed. It carries a real cost: someone must define the overlap period and decide how consumers avoid treating old and translated events as separate business actions. Renaming a topic does not settle that question.
Keep the strongest alternative in the decision record. A translating reader can normalize historical records without maintaining two live streams. It is attractive when the translation is deterministic and preserves the original meaning. It becomes a poor boundary when it must call mutable business services to guess missing facts. The same replay could then produce different answers on different days.
For either route, test the rollback reader against messages produced after rollout. If it cannot read them, rollback requires a compatible reader release or an explicit containment procedure. Returning to an old binary is not sufficient evidence of reversibility.
Record the decision and its expiry conditions
The decision worksheet asks for the supported writer history, allowed readers and an example of each semantic change. It also separates three release gates: decoding succeeds, the interpreted fact is correct and the downstream business action remains permitted. The worksheet is a proposal for a review meeting, not a completed audit of an application.
Assign ownership of the replay boundary to the team that operates the consuming behavior. A registry owner can enforce a compatibility policy, but cannot decide whether unknown is acceptable for a customer's routing decision. Keep both approvals visible when those responsibilities belong to different teams.
Revisit the choice when the replay horizon grows, an external consumer gains independent release control or a new field becomes mandatory for an irreversible action. Before the next producer change, put the oldest supported message in front of the rollback reader and record its interpreted values, not only its exit code.
Sources
Documentation checked .