Concept Library

Schema Evolution and Backwards Compatibility

How to change the shape of data over time — adding columns, renaming fields, deprecating tables — without breaking the consumers downstream.

Data Engineering
Intermediate
7 min
Last reviewed 2026-Q2

What it is

Schema evolution is the management of change. Tables grow columns. Event payloads gain fields. Types narrow or widen. A string becomes an enum. A field is renamed. A table is deprecated.

Backwards compatibility means consumers built against the old schema continue to work against the new one. Forwards compatibility means producers using the new schema produce data that consumers built against the old schema can still read. Full compatibility — both directions — is the gold standard for event-driven systems.

The hard part is not the engineering. It is the discipline of treating schemas as long-lived contracts and changing them deliberately.

Why it matters

Most production data incidents trace back to an unannounced schema change. A column rename. A type narrowing. A new required field. The producer ships, the consumer breaks, the dashboard goes red.

Schema evolution is also the tax on speed. Teams that cannot evolve their schemas safely either ship slowly (every change requires a global review) or ship recklessly (every change is a coin flip on whether something downstream breaks). Teams that have invested in evolution practices can ship daily without fear.

For event-driven systems and streaming platforms, schema evolution is non-negotiable. A misaligned schema in a Kafka topic does not just break a dashboard — it can poison every consumer for hours.

How it works

The compatibility taxonomy is consistent across formats and registries.

Backwards compatible changes — old consumers can read new data:

Adding an optional field
Adding a default value to an existing field
Removing a field that consumers do not use

Forwards compatible changes — new consumers can read old data:

Removing a required field (with default)
Adding a required field (with default)

Breaking changes — require coordination:

Renaming a field
Changing a field’s type incompatibly
Removing a field still in use
Tightening constraints (nullable to non-nullable)

For Avro / Protobuf / JSON Schema, the registry enforces compatibility per topic. For warehouses, dbt model contracts and Iceberg’s schema evolution rules play a similar role.

The strongest pattern for breaking changes is the parallel pattern: introduce the new field alongside the old, migrate consumers one at a time, deprecate the old, remove the old. Slower than a hard cutover, dramatically safer.

Vendor comparison

Context	Tool	Behavior
Event-driven (Kafka)	Confluent Schema Registry, Apicurio	Enforces compatibility on every published schema; producers fail to register incompatible schemas
Iceberg tables	Iceberg native	Add column, drop column, rename column all without rewriting data; type widening allowed
Delta tables	Delta native	Add column, type widening; column rename via column mapping mode
Hudi tables	Hudi native	Schema evolution supported with some limits on column drops
dbt model contracts	dbt	Validates contract on materialization; breaks build on incompatible change
SQLMesh	SQLMesh	Column-level lineage + virtual environments make breaking changes testable before deploy

Iceberg has the strongest schema evolution story among open table formats — full support for add, drop, rename, type widening, all without rewriting underlying data files. This is one of the reasons it has become the lakehouse default.

Delta and Hudi are close but with edge cases. Delta’s column mapping mode is required for renames; Hudi has some restrictions on certain combinations.

For event streaming, Confluent Schema Registry remains the practical default. Apicurio is the open-source alternative.

Yoann’s take

The discipline I enforce: every schema change is a pull request, every breaking change is parallel-pattern migration, every consumer-facing schema has a contract in the producer’s repo. The tooling helps but does not substitute for the discipline. I have seen teams with Schema Registry deployed and ignored — producers find ways around it because nobody enforces the operating model. The unlock is making schema discipline part of code review culture, not a tool you install. On lakehouse platforms my default is Iceberg specifically because its schema evolution model is the most forgiving, which lowers the cost of mistakes.

— Yoann

External references