Published on

The Evolving Contract: Mastering Event Schema Evolution with Kafka, Avro, and Spring Boot

Authors
  • avatar
    Name
    Maria
    Twitter

Introduction

In the vibrant world of event-driven architectures, Apache Kafka stands as a cornerstone for building scalable and resilient microservices. Events flowing through Kafka streams represent the very language of our domain, carrying crucial data between services. But herein lies a critical, often underestimated, challenge: What happens when the structure of these events needs to change?

Imagine you've deployed a core OrderPlaced event. Over time, business requirements evolve, and you need to add a shippingAddress field or perhaps rename customerZipCode to postalCode. Without a robust strategy, such changes can swiftly turn into a production nightmare. Older consumers might fail to deserialize new events, leading to data loss, service disruptions, or even cascading failures across your entire system. This isn't just a development inconvenience; it's a fundamental architectural contract problem. How do we evolve our event schemas gracefully, ensuring both backward and forward compatibility without bringing down our production environment?

This deep dive will equip you with the knowledge and tools to master event schema evolution using Apache Avro, Confluent Schema Registry, and Spring Boot. We'll explore the core concepts, demonstrate practical implementation, and discuss the critical considerations to keep your event-driven systems robust and continuously evolving.

Deep Dive: The Unseen Contract – Avro and Schema Registry

The solution to graceful schema evolution lies in establishing a clear, enforceable contract for your event data, managed centrally. This is where Apache Avro and Confluent Schema Registry shine.

Apache Avro: Data Serialization with a Self-Describing Schema

Avro is a language-agnostic data serialization system that distinguishes itself by requiring schemas during both data writing and reading. Unlike JSON, which relies on implicit understanding, or Protocol Buffers, which uses compiled IDL files, Avro embeds the schema directly with the data or references a schema by ID. This "schema-on-read" approach is incredibly powerful for evolution.

Key features of Avro for schema evolution:

  • Compact Binary Format: Efficient for storage and network transfer.
  • Rich Data Types: Supports primitives, records, enums, arrays, maps, unions.
  • Schema-on-Read: The data itself doesn't contain field names, only values. The reader's schema dictates how to interpret the binary data.
  • Schema Resolution: When reading data, Avro attempts to resolve the writer's schema against the reader's schema. It can automatically handle certain discrepancies, such as:
    • Adding fields (if they have default values or are nullable in the reader's schema).
    • Removing fields (if the reader's schema defines a default or omits them).
    • Renaming fields (if aliasing is used).

Confluent Schema Registry: The Central Authority for Your Data Contracts

While Avro provides the mechanism, the Confluent Schema Registry provides the management. It's a distributed storage layer for schemas that uses Kafka as its durable backend. Its primary functions are:

  1. Centralized Schema Storage: All schemas for Kafka topics are stored in a single, accessible location.
  2. Schema ID Generation: Each registered schema is assigned a unique, monotonically increasing ID. When an Avro message is produced, only this ID (or the schema itself for the first message on a topic) needs to be sent with the payload, reducing message size.
  3. Compatibility Checks: This is the game-changer. For every new schema version registered for a given subject (typically a Kafka topic name suffixed with -value or -key), the Schema Registry enforces a configured compatibility rule.

Understanding Compatibility Rules

Compatibility rules define how a new schema version relates to previous ones. Choosing the right rule is paramount:

  • NONE (Default): No compatibility checks. Highly dangerous for production.
  • BACKWARD: A new schema can be used by older consumers to read data written with the new schema. This means new code can read old data, and old code can read new data.
    • Allowed changes: Adding an optional field (with a default), removing a field (if it had a default in old schema).
    • Implication: Ensures existing consumers don't break when a producer rolls out a new schema. This is often the safest default.
  • FORWARD: A new schema can be used to read data written with an older schema.
    • Allowed changes: Adding required fields (old data will implicitly map to null or default), removing optional fields.
    • Implication: Ensures new consumers can process old data.
  • FULL: A schema is FULL compatible if it's both BACKWARD and FORWARD compatible. This is the most restrictive but offers the highest guarantee of compatibility.
    • Allowed changes: Only adding optional fields with defaults.
  • _TRANSITIVE variants: These apply the compatibility check not just against the immediate previous schema version, but against all prior schema versions. This offers stricter guarantees but can limit evolution flexibility. BACKWARD_TRANSITIVE is often recommended for maximum safety.

How it works in practice:

  1. Producer: Serializes an event using its current schema. Before sending, it registers or retrieves the schema ID from Schema Registry. The message sent to Kafka contains the schema ID and the binary Avro data.
  2. Consumer: Receives the message, extracts the schema ID, and retrieves the corresponding writer's schema from Schema Registry. It then uses its own reader's schema (compiled into the consumer application) to deserialize the message according to Avro's schema resolution rules.

Code Implementation: Spring Boot with Avro and Schema Registry

Let's walk through a practical example using Spring Boot, focusing on a simple OrderCreated event.

1. Project Setup (Producer & Consumer)

We'll use Maven for our build. Add the following to your pom.xml:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>4.0.0-M1</version> <!-- Assuming Spring Boot 4.0 -->
        <relativePath/> <!-- lookup parent from repository -->
    </parent>
    <groupId>com.example</groupId>
    <artifactId>kafka-avro-schema-evolution</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <name>kafka-avro-schema-evolution</name>
    <description>Demo for Kafka Avro Schema Evolution with Spring Boot</description>

    <properties>
        <java.version>21</java.version> <!-- Use Java 21 for Spring Boot 4.0, or 25 if available -->
        <avro.version>1.11.1</avro.version>
        <confluent.version>7.5.0</confluent.version> <!-- Match your Schema Registry version -->
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.kafka</groupId>
            <artifactId>spring-kafka</artifactId>
        </dependency>
        <dependency>
            <groupId>org.apache.avro</groupId>
            <artifactId>avro</artifactId>
            <version>${avro.version}</version>
        </dependency>
        <dependency>
            <groupId>io.confluent</groupId>
            <artifactId>kafka-avro-serializer</artifactId>
            <version>${confluent.version}</version>
        </dependency>

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.springframework.kafka</groupId>
            <artifactId>spring-kafka-test</artifactId>
            <scope>test</scope>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
            <plugin>
                <groupId>org.apache.avro</groupId>
                <artifactId>avro-maven-plugin</artifactId>
                <version>${avro.version}</version>
                <executions>
                    <execution>
                        <phase>generate-sources</phase>
                        <goals>
                            <goal>schema</goal>
                        </goals>
                        <configuration>
                            <sourceDirectory>${project.basedir}/src/main/resources/avro/</sourceDirectory>
                            <outputDirectory>${project.basedir}/src/main/java/</outputDirectory>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>

    <repositories>
        <repository>
            <id>spring-milestones</id>
            <name>Spring Milestones</name>
            <url>https://repo.spring.io/milestone</url>
            <snapshots>
                <enabled>false</enabled>
            </snapshots>
        </repository>
        <repository>
            <id>confluent</id>
            <url>https://packages.confluent.io/maven/</url>
        </repository>
    </repositories>

</project>

Note on Java 25 & Spring Boot 4.0: As Spring Boot 4.0 is still in preview and Java 25 isn't officially released, I've used Java 21 and 4.0.0-M1 as a placeholder for the parent, which would be the likely compatible version. The core Kafka/Avro integration principles remain the same.

2. Kafka and Schema Registry Setup (Docker Compose)

To run this, you'll need Kafka and Schema Registry. A docker-compose.yml file is ideal:

version: '3.8'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:7.5.0
    hostname: zookeeper
    container_name: zookeeper
    ports:
      - '2181:2181'
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

  broker:
    image: confluentinc/cp-kafka:7.5.0
    hostname: broker
    container_name: broker
    ports:
      - '9092:9092'
      - '9093:9093'
    depends_on:
      - zookeeper
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:9093,PLAINTEXT_HOST://localhost:9092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1

  schema-registry:
    image: confluentinc/cp-schema-registry:7.5.0
    hostname: schema-registry
    container_name: schema-registry
    ports:
      - '8081:8081'
    depends_on:
      - broker
    environment:
      SCHEMA_REGISTRY_HOST_NAME: schema-registry
      SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: 'broker:9093'
      SCHEMA_REGISTRY_LISTENERS: http://0.0.0.0:8081

Run docker compose up -d to start these services.

3. Defining the Avro Schema (V1)

Create src/main/resources/avro/OrderCreatedEventV1.avsc:

{
  "namespace": "com.example.events",
  "name": "OrderCreatedEvent",
  "type": "record",
  "fields": [
    {
      "name": "orderId",
      "type": "string"
    },
    {
      "name": "productId",
      "type": "string"
    },
    {
      "name": "quantity",
      "type": "int"
    },
    {
      "name": "customerEmail",
      "type": "string"
    }
  ]
}

Run mvn clean generate-sources to generate the OrderCreatedEvent Java class.

4. Spring Boot Configuration

application.yml:

spring:
  application:
    name: kafka-avro-schema-evolution
  kafka:
    producer:
      bootstrap-servers: localhost:9092
      key-serializer: org.apache.kafka.common.serialization.StringSerializer
      value-serializer: io.confluent.kafka.serializers.KafkaAvroSerializer
      properties:
        schema.registry.url: http://localhost:8081
    consumer:
      bootstrap-servers: localhost:9092
      group-id: order-consumers
      auto-offset-reset: earliest
      key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
      value-deserializer: io.confluent.kafka.serializers.KafkaAvroDeserializer
      properties:
        schema.registry.url: http://localhost:8081
        specific.avro.reader: true # Important: tells Avro to use specific generated classes

5. The Producer (V1)

// src/main/java/com/example/producer/OrderProducerV1.java
package com.example.producer;

import com.example.events.OrderCreatedEvent;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.stereotype.Service;

import java.util.UUID;

@Service
public class OrderProducerV1 {

    private static final String TOPIC = "order-events";
    private final KafkaTemplate<String, OrderCreatedEvent> kafkaTemplate;

    public OrderProducerV1(KafkaTemplate<String, OrderCreatedEvent> kafkaTemplate) {
        this.kafkaTemplate = kafkaTemplate;
    }

    public void sendOrderCreatedEvent() {
        String orderId = UUID.randomUUID().toString();
        OrderCreatedEvent event = OrderCreatedEvent.newBuilder()
                .setOrderId(orderId)
                .setProductId("PROD-" + UUID.randomUUID().toString().substring(0, 4))
                .setQuantity(1)
                .setCustomerEmail("customer-" + orderId.substring(0, 4) + "@example.com")
                .build();

        kafkaTemplate.send(TOPIC, orderId, event);
        System.out.println("V1 Producer sent OrderCreatedEvent: " + event);
    }
}

6. The Consumer (V1)

// src/main/java/com/example/consumer/OrderConsumerV1.java
package com.example.consumer;

import com.example.events.OrderCreatedEvent;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.stereotype.Service;

@Service
public class OrderConsumerV1 {

    @KafkaListener(topics = "order-events", groupId = "order-consumers-v1")
    public void consume(OrderCreatedEvent event) {
        System.out.println("V1 Consumer received OrderCreatedEvent: " + event);
        // Simulate processing the event
        System.out.println("    Order ID: " + event.getOrderId());
        System.out.println("    Product ID: " + event.getProductId());
        System.out.println("    Quantity: " + event.getQuantity());
        System.out.println("    Customer Email: " + event.getCustomerEmail());
    }
}

7. Initial Test (V1 Producer & V1 Consumer)

Run your Spring Boot application with both producer and consumer active. The producer will send OrderCreatedEventV1, and the consumer will successfully read it. Check the Schema Registry UI (if you have one, e.g., using confluentinc/cp-schema-registry and a tool like akhq or Lenses) or curl http://localhost:8081/subjects/order-events-value/versions/latest to see your V1 schema registered.

8. Schema Evolution: Introducing V2 (Backward Compatible)

Now, let's evolve the schema. We want to add an optional orderDate field.

Update src/main/resources/avro/OrderCreatedEventV1.avsc (rename it or create a new file, but for simplicity we'll overwrite and generate a new class):

{
  "namespace": "com.example.events",
  "name": "OrderCreatedEvent",
  "type": "record",
  "fields": [
    {
      "name": "orderId",
      "type": "string"
    },
    {
      "name": "productId",
      "type": "string"
    },
    {
      "name": "quantity",
      "type": "int"
    },
    {
      "name": "customerEmail",
      "type": "string"
    },
    {
      "name": "orderDate",
      "type": ["null", "long"],
      "default": null,
      "doc": "Timestamp of when the order was placed (epoch milliseconds)"
    }
  ]
}

Important: We added orderDate with type: ["null", "long"] and "default": null. This makes it an optional field. This change is backward compatible.

Run mvn clean generate-sources again to update the OrderCreatedEvent Java class.

9. The Producer (V2)

Update your producer to send the new field:

// src/main/java/com/example/producer/OrderProducerV2.java (or update V1 producer)
package com.example.producer;

import com.example.events.OrderCreatedEvent;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.stereotype.Service;

import java.time.Instant;
import java.util.UUID;

@Service
public class OrderProducerV2 { // Renamed for clarity, could be the same class

    private static final String TOPIC = "order-events";
    private final KafkaTemplate<String, OrderCreatedEvent> kafkaTemplate;

    public OrderProducerV2(KafkaTemplate<String, OrderCreatedEvent> kafkaTemplate) {
        this.kafkaTemplate = kafkaTemplate;
    }

    public void sendOrderCreatedEvent() {
        String orderId = UUID.randomUUID().toString();
        OrderCreatedEvent event = OrderCreatedEvent.newBuilder()
                .setOrderId(orderId)
                .setProductId("PROD-" + UUID.randomUUID().toString().substring(0, 4))
                .setQuantity(1)
                .setCustomerEmail("customer-" + orderId.substring(0, 4) + "@example.com")
                .setOrderDate(Instant.now().toEpochMilli()) // Set the new field
                .build();

        kafkaTemplate.send(TOPIC, orderId, event);
        System.out.println("V2 Producer sent OrderCreatedEvent: " + event);
    }
}

10. The Consumer (V2)

First, keep the V1 Consumer running. Then, deploy a new V2 Consumer (or update the V1 consumer code).

// src/main/java/com/example/consumer/OrderConsumerV2.java
package com.example.consumer;

import com.example.events.OrderCreatedEvent;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.stereotype.Service;

import java.time.Instant;
import java.util.Optional;

@Service
public class OrderConsumerV2 {

    @KafkaListener(topics = "order-events", groupId = "order-consumers-v2")
    public void consume(OrderCreatedEvent event) {
        System.out.println("V2 Consumer received OrderCreatedEvent: " + event);
        System.out.println("    Order ID: " + event.getOrderId());
        System.out.println("    Product ID: " + event.getProductId());
        System.out.println("    Quantity: " + event.getQuantity());
        System.out.println("    Customer Email: " + event.getCustomerEmail());

        // Handle the new optional field
        Optional.ofNullable(event.getOrderDate())
                .map(Instant::ofEpochMilli)
                .ifPresentOrElse(
                        date -> System.out.println("    Order Date: " + date),
                        () -> System.out.println("    Order Date: N/A (older event)")
                );
    }
}

11. Testing Schema Evolution

  1. Start your docker-compose.
  2. Deploy the V1 Producer and V1 Consumer. Send some events. The V1 consumer should process them fine.
  3. Crucially: Now, deploy the V2 Producer. This will register a new schema version (V2) with the Schema Registry. Since we made a backward-compatible change, the Schema Registry will allow it.
    • The V1 Consumer will still be able to read events produced by the V2 Producer! The Avro deserializer, using the V1 schema, will simply ignore the new orderDate field.
    • The V1 Producer can continue to send V1 events (if you have old producers still running), and the V2 Consumer will read them. For the orderDate field, it will see null because the V1 event didn't contain it, demonstrating forward compatibility from the V2 consumer's perspective.
  4. Deploy the V2 Consumer. It will correctly read events from both V1 Producer (where orderDate will be null) and V2 Producer (where orderDate will be present).

This demonstrates the power of BACKWARD compatibility (V1 Consumer reading V2 data) and FORWARD compatibility (V2 Consumer reading V1 data when the field is optional/has default). To enforce BACKWARD_TRANSITIVE on your Schema Registry for the topic order-events-value, you'd typically use curl -X PUT -H "Content-Type: application/vnd.schemaregistry.v1+json" --data '{"compatibility": "BACKWARD_TRANSITIVE"}' http://localhost:8081/config/order-events-value.

Considerations and Trade-offs

While Avro and Schema Registry provide a robust solution, there are important factors to consider for production environments:

  • Complexity: Introducing Avro and Schema Registry adds another layer of tooling and configuration. Teams need to understand Avro schemas, the avro-maven-plugin (or Gradle equivalent), and compatibility rules.
  • Performance Overhead: Avro serialization/deserialization is generally efficient, but it does incur a slight overhead compared to raw byte arrays or highly optimized custom serializers. For the vast majority of applications, this is negligible.
  • Schema Registry Availability: The Schema Registry is a critical component. If it's down, producers cannot register new schemas or look up existing ones, and consumers cannot retrieve writer schemas, potentially halting your event flow. Ensure it's deployed in a highly available, clustered configuration.
  • Managing Schema Files (.avsc): Treat your .avsc files like source code. Version control them alongside your application code. Ensure your CI/CD pipeline correctly generates Avro classes and validates schema compatibility during development and deployment.
  • Choosing the Right Compatibility Mode: BACKWARD_TRANSITIVE is often recommended as it offers the highest safety, ensuring no current or future consumer will break. However, it can be restrictive. Understand the implications of each mode for your specific use cases. FULL_TRANSITIVE is even more restrictive, only allowing optional field additions with defaults.
  • Breaking Changes & Radical Schema Revisions: Sometimes, a change is fundamentally incompatible (e.g., removing a required field without a default, changing a field's type drastically). In such cases:
    • New Topic: Create a new Kafka topic for the new event version (order-events-v2).
    • Parallel Deployment: Deploy new producers to the new topic and new consumers to both topics (old and new).
    • Data Migration: If historical data needs to be compatible, you might need a one-time migration process to transform old events to the new schema and write them to the new topic.
    • Sunset Old Topic: Gradually transition all producers and consumers to the new topic, then decommission the old one.
  • Schema Registry UI: Tools like Lenses.io or Confluent Control Center provide excellent UIs for browsing schemas, their versions, and compatibility settings. These are invaluable for operations and debugging.

Conclusion

Event schema evolution is not a peripheral concern; it's a core architectural challenge that demands a robust solution in event-driven microservices. By leveraging Apache Avro for its self-describing, binary serialization capabilities and the Confluent Schema Registry for centralized management and compatibility enforcement, you can gracefully evolve your data contracts.

We've seen how Spring Boot seamlessly integrates with these tools, allowing you to produce and consume events with confidence, even as their underlying structure changes. Embracing this pattern ensures that your systems remain adaptable, resilient, and continuously deliver value without the fear of breaking changes. Master your event contracts, and you master the heart of your event-driven architecture.