The Hidden Complexity of Schema Evolution: How Data Outlives the Code That Created It
A theoretical framework for thinking about backward and forward compatibility across Avro, Protobuf, JSON, and relational schemas — before a breaking change reaches production.
Code gets deployed, rolled back, and eventually deleted. Data, on the other hand, tends to stick around. A message written to Kafka today might still be read by a consumer three years from now, long after the service that produced it has been rewritten twice. That mismatch — short-lived code reading long-lived data — is the real source of almost every schema evolution headache. It is not a formatting problem. It is a time problem.
This piece walks through a way of thinking about schema change that applies whether you are working with Avro, Protobuf, JSON, or a plain relational table. The goal is not to memorize rules for each format, but to build a mental model so that when a new format shows up, you already know which questions to ask.
Why Schema Evolution Feels Simple Until It Isn’t
Adding a field feels harmless. Renaming one feels almost cosmetic. Removing a field that “nothing uses anymore” feels like cleanup. And yet, each of these has caused production incidents at companies with excellent engineers. The reason is that a schema is not really owned by the writer or the reader — it is a contract between two parties that rarely deploy at the same time.
Consider a producer service upgraded to version 2 of a schema while five consumer services are still running version 1. For a window that can last hours or weeks, both versions of the data exist in the wild simultaneously. If the format does not define what happens in that window, someone finds out the hard way, usually through a null-pointer exception or a silently dropped record.
Backward, Forward, and Full Compatibility — A Shared Vocabulary
Before comparing formats, it helps to agree on what “compatible” actually means. The Confluent Schema Registry documentation frames this cleanly, and the same definitions apply well beyond Kafka.
| Compatibility type | What it guarantees | Typical use case |
|---|---|---|
| Backward | New schema can read data written with the old schema | Upgrading consumers first |
| Forward | Old schema can read data written with the new schema | Upgrading producers first |
| Full | Both directions hold at once | Rolling deployments with no fixed order |
| None | No guarantee; every change is a coordinated release | Tightly coupled internal systems |
Most teams instinctively want “full” compatibility everywhere, but that ambition quietly rules out some genuinely useful changes, such as making a previously optional field required. Knowing which guarantee you actually need for a given data stream is half the battle.
How Each Format Actually Handles Change
Avro
Apache Avro ties every message to a writer schema and a reader schema, then resolves differences between the two at read time. This is why Avro leans so heavily on default values: as long as a new field has a default, old data can be read as if that field had always existed. Field aliases also let you rename a field without breaking anyone who still refers to it by the old name.
Protobuf
Protocol Buffers takes a different route: compatibility is tracked through numbered fields rather than field names. This is genuinely elegant, because renaming a field in the .proto file changes nothing about the binary encoding. The danger moves elsewhere — reusing a retired field number for something new is the classic Protobuf mistake, since old binary data will silently be reinterpreted under the new meaning.
JSON Schema
Plain JSON and JSON Schema have no built-in resolution mechanism at all. Compatibility depends entirely on convention: whether consumers ignore unknown fields, whether `additionalProperties` is set, and whether required fields were chosen conservatively from day one. This flexibility is exactly what makes JSON so easy to start with, and exactly what makes it easy to break later, since nothing enforces the contract for you.
Relational Schemas
Relational databases evolve through explicit migrations rather than per-message negotiation. A column addition with a default value is usually safe; dropping or renaming a column tends to be the risky operation, because every query, view, and downstream ETL job that references the old name breaks at once. Tools like Flyway and Liquibase formalize this into versioned, ordered steps, which is effectively a manual version of what Avro and Protobuf try to automate.
Where the Failures Actually Cluster
In practice, two categories of change cause the overwhelming majority of incidents: renames and type changes. Everything else — adding optional fields, deprecating with a grace period, appending enum values — tends to be manageable if the format supports it at all. Renames are dangerous because they look free (the data hasn’t changed shape) while actually breaking any consumer matching on field names. Type changes are dangerous because narrowing a type, say from a 64-bit integer to a 32-bit one, can silently truncate data rather than fail loudly.
Notice that no format wins across the board. Protobuf and Avro trade blows on tooling maturity versus human readability, while JSON wins on approachability and loses on built-in guarantees. Relational schemas sit in the middle because migration tooling has matured enormously, even though the underlying mechanism is still manual.
A Practical Checklist Before You Ship a Schema Change
Regardless of format, most breaking changes can be caught before deployment by asking a short set of questions. This is less about tooling and more about discipline.
Run this checklist before merging a schema change:
- Does the new field have a default value, or is it truly optional for every existing consumer?
- If a field is being removed, has it been deprecated long enough that no active consumer still reads it?
- If a field is being renamed, is there an alias or a transition period rather than a hard cutover?
- Does the type change widen the representable range, rather than narrow it?
- Has the change been validated against a compatibility check (schema registry, contract test, or migration dry run) rather than assumed safe?
That last point is worth emphasizing. Tools like the Confluent Schema Registry, Buf’s breaking change detector for Protobuf, and JSON Schema diff tools all exist precisely because human review tends to miss subtle incompatibilities, especially under deadline pressure.
What We Learned
Schema evolution is not really about syntax — it’s about the fact that producers and consumers rarely upgrade in lockstep, and data written today may be read by code written years from now. Avro and Protobuf bake compatibility guarantees into their design, using default values and numbered fields respectively, while JSON and relational schemas rely more on convention and disciplined migrations. Across all four, the same two operations cause almost every real incident: renaming fields and narrowing types. A short pre-deployment checklist, backed by an actual compatibility check rather than a guess, closes most of that gap regardless of which format you’re running.
