Resilient Kafka Consumers with Spring Boot: Retries, DLTs, Idempotency, and Recovery

A Kafka consumer is resilient when it can make progress without losing business intent, duplicating irreversible effects, or hiding failed records. That requires more than adding three retries and a dead-letter topic.

The hard part is deciding what each failure means:

• Is it transient, permanent, or caused by bad data?
• Is retrying safe?
• Should retry preserve partition order?
• Has an external side effect already happened?
• Who can repair and replay the record?
• How will an operator prove that the replay did not apply the effect twice?

This guide designs a Spring Boot 4.1 and Spring Kafka consumer around those questions. The example processes orders.created.v1 records, writes an idempotent database effect, retries only transient failures, and routes exhausted records to a dead-letter topic with enough context for investigation.

TL;DR Assume at-least-once delivery and make business effects idempotent. Classify exceptions before choosing a retry policy. Keep blocking retries short; use retry topics for long delays when reordering is acceptable. A Kafka transaction can atomically commit consumed offsets and produced Kafka records, but it does not automatically make a database update or HTTP call part of that transaction. A dead-letter topic is a recovery queue with ownership, retention, alerts, and a replay procedure—not a place where failures disappear.

Begin with the Processing Contract

Define what success means before configuring a listener:

Input:
  topic = orders.created.v1
  key = orderId
  eventId = globally unique and immutable

Success:
  inventory reservation exists for eventId
  audit row exists
  input offset can advance

Duplicate:
  same eventId produces no additional reservation

Transient failure:
  bounded retry with backoff

Permanent or exhausted failure:
  original record and failure context are published to a DLT

Recovery:
  operator repairs the cause and replays through the same idempotent path

The event identifier is part of the contract. Kafka's topic, partition, and offset identify a log position, but a business event may be republished to another topic or restored from a backup. A stable event ID remains useful across those movements.

Understand the Four Failure Windows

With an ordinary at-least-once listener, four points matter:

1. The record is fetched.
2. Business processing begins.
3. Database or external effects complete.
4. The consumer offset is committed.

If the process stops after step 3 and before step 4, Kafka can deliver the record again. That is expected behavior, not a broker bug.

Kafka producer idempotence prevents duplicate log entries caused by a producer retry in its supported scope. Kafka transactions can make consumed offsets and newly produced Kafka records atomic. Neither feature automatically rolls an ordinary PostgreSQL update or a payment-provider API call into the Kafka transaction.

Use the right consistency tool for the boundary:

Avoid presenting a KafkaTransactionManager and a JPA transaction manager as if they create a portable, failure-free distributed transaction. Commit ordering and crash windows still need analysis.

Create an Event Envelope That Supports Recovery

An envelope should carry stable identity and schema information without turning every record into an unbounded metadata dump:

public record OrderCreated(
    UUID eventId,
    int schemaVersion,
    Instant occurredAt,
    String orderId,
    String customerId,
    List<OrderLine> lines
) {
}

Validate structural requirements at the boundary:

static void validate(OrderCreated event) {
    if (event.eventId() == null) {
        throw new InvalidOrderEvent("eventId is required");
    }
    if (event.schemaVersion() != 1) {
        throw new UnsupportedOrderSchema(
            "Unsupported schema version: " + event.schemaVersion()
        );
    }
    if (event.lines() == null || event.lines().isEmpty()) {
        throw new InvalidOrderEvent("At least one order line is required");
    }
}

Schema incompatibility and invalid business data will not become valid after a two-second sleep. Classify them as non-retryable.

Make the Database Effect Idempotent

Use a unique database constraint as the final concurrency guard:

create table processed_event (
    consumer_name varchar(100) not null,
    event_id uuid not null,
    processed_at timestamptz not null,
    primary key (consumer_name, event_id)
);

Then claim and apply the event in the same database transaction:

@Service
public class OrderCreatedHandler {

    private static final String CONSUMER = "inventory-reservation-v1";

    private final ProcessedEventRepository processedEvents;
    private final InventoryRepository inventory;

    public OrderCreatedHandler(
            ProcessedEventRepository processedEvents,
            InventoryRepository inventory) {
        this.processedEvents = processedEvents;
        this.inventory = inventory;
    }

    @Transactional
    public ProcessingResult handle(OrderCreated event) {
        validate(event);

        boolean claimed = processedEvents.tryInsert(
            CONSUMER,
            event.eventId(),
            Instant.now()
        );

        if (!claimed) {
            return ProcessingResult.DUPLICATE;
        }

        for (OrderLine line : event.lines()) {
            inventory.reserve(
                event.orderId(),
                line.productId(),
                line.quantity()
            );
        }

        return ProcessingResult.APPLIED;
    }
}

tryInsert should use an atomic insert such as PostgreSQL INSERT ... ON CONFLICT DO NOTHING and return whether it inserted a row. A prior SELECT followed by INSERT has a race between concurrent consumers or replays.

Because the processed marker and inventory update share one database transaction, a rollback removes both. If the database commits and the process dies before the Kafka offset commit, redelivery finds the marker and produces no second reservation.

For a long-running handler, do not keep a database transaction open while calling an unreliable external API. Persist an intent, commit it, and let another recoverable step perform the call using an idempotency key.

Classify Failures Explicitly

A useful taxonomy is:

Do not catch Exception inside the listener and return normally. The container interprets a normal return as successful processing and can commit the offset.

Do not retry every exception indefinitely. One poison record can block a partition forever and create growing lag.

Configure Bounded Blocking Retries

Spring Kafka's DefaultErrorHandler can seek and redeliver a failed record. A DeadLetterPublishingRecoverer publishes the record after retries are exhausted.

import org.apache.kafka.common.TopicPartition;
import org.springframework.context.annotation.Bean;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.kafka.listener.DeadLetterPublishingRecoverer;
import org.springframework.kafka.listener.DefaultErrorHandler;
import org.springframework.util.backoff.ExponentialBackOffWithMaxRetries;

@Bean
DefaultErrorHandler orderErrorHandler(
        KafkaTemplate<Object, Object> template) {

    DeadLetterPublishingRecoverer recoverer =
        new DeadLetterPublishingRecoverer(
            template,
            (record, exception) ->
                new TopicPartition(
                    record.topic() + ".dlt",
                    record.partition()
                )
        );

    ExponentialBackOffWithMaxRetries backOff =
        new ExponentialBackOffWithMaxRetries(3);
    backOff.setInitialInterval(500L);
    backOff.setMultiplier(2.0);
    backOff.setMaxInterval(5_000L);

    DefaultErrorHandler handler =
        new DefaultErrorHandler(recoverer, backOff);

    handler.addNotRetryableExceptions(
        InvalidOrderEvent.class,
        UnsupportedOrderSchema.class
    );

    return handler;
}

Attach it to the listener container factory:

@Bean
ConcurrentKafkaListenerContainerFactory<String, OrderCreated>
orderKafkaListenerContainerFactory(
        ConsumerFactory<String, OrderCreated> consumerFactory,
        DefaultErrorHandler orderErrorHandler) {

    var factory =
        new ConcurrentKafkaListenerContainerFactory<String, OrderCreated>();
    factory.setConsumerFactory(consumerFactory);
    factory.setCommonErrorHandler(orderErrorHandler);
    return factory;
}

And keep the listener thin:

@KafkaListener(
    topics = "orders.created.v1",
    groupId = "inventory-reservation-v1",
    containerFactory = "orderKafkaListenerContainerFactory"
)
public void onOrderCreated(OrderCreated event) {
    handler.handle(event);
}

Blocking retries pause delivery for the affected consumer thread. They are appropriate for a few short attempts. They are a poor fit for a thirty-minute dependency outage because they consume consumer capacity and may interact badly with poll intervals and rebalances.

Spring Kafka's exception-handling reference is the source of truth for handler behavior and batch-listener differences.

Decide Whether Ordering Matters

If all events for an order use orderId as their key, Kafka keeps those records in one partition and the consumer observes partition order. Moving a failed event to retry topics allows later records from the source partition to progress, so retries can reorder business events.

Choose deliberately:

• Use short blocking retries when strict per-key order matters and the outage is expected to be brief.
• Use retry topics when delays are long and the business can tolerate reordering or enforce versions.
• Stop or pause processing when applying a later event before the failed event would corrupt state.
• Include an aggregate version when stale or out-of-order updates must be rejected.

@RetryableTopic is useful for non-blocking retry topologies, but it is not a universal replacement for container error handling. Check listener type, ordering requirements, number of generated topics, retention, and operational visibility.

Design the Dead-Letter Topic for Recovery

A DLT should preserve:

• original topic, partition, offset, timestamp, and key;
• original value or an approved redacted representation;
• exception class and a bounded error message;
• consumer group or logical consumer name;
• retry count and first/last failure time;
• application version and trace or correlation identifier;
• schema version.

Spring's recoverer adds standard original-record and exception headers. Verify header size and privacy before relying on defaults. Broker record-size limits still apply, and stack traces can expose sensitive details.

Create an operational policy:

Owner: Inventory team
Alert: any DLT record, grouped to avoid alert storms
Retention: 14 days
Triage target: 4 business hours
Replay approval: one engineer for data repair, two for financial effects
Replay destination: source topic through a controlled tool
Audit: operator, reason, record count, source offsets, target, result

Do not run a permanent consumer that automatically copies every DLT record straight back to the source. A persistent poison record will form a hot loop.

Build a Safe Replay Procedure

Recovery is part of the feature, not an afterthought:

1. Stop the replay if the underlying defect or data problem is not understood.
2. Export the affected record identifiers and original coordinates.
3. Fix the consumer, dependency, or data through an approved process.
4. Confirm that the consumer's effect is idempotent.
5. Replay a single record in a non-production or shadow path where possible.
6. Replay a bounded batch with rate limiting.
7. Compare processed, duplicate, rejected, and failed counts.
8. Record the operator and reason.

Do not edit a signed or auditable business event silently. If repair requires transformation, preserve the original and record the transformation as a new, traceable artifact.

Kafka Transactions: Use Them Within Their Boundary

For a read-process-write pipeline whose input and outputs all live in Kafka, a Kafka transaction can atomically publish output records and commit the consumed offsets. Downstream consumers must use isolation.level=read_committed to hide aborted records.

This provides a strong Kafka-to-Kafka guarantee. It does not guarantee that:

• an email was sent once;
• a payment provider charged once;
• a PostgreSQL update committed once;
• a non-transactional search index was updated once.

Those effects need their own idempotency, durable intent, or reconciliation strategy. See the site's transactional outbox guide when a database change must reliably cause a Kafka publication, and the exactly-once processing guide for a deeper discussion of guarantee boundaries.

Observe the System You Need to Recover

Track at least:

• consumer lag by group, topic, and partition;
• processing latency and throughput;
• retry attempts by exception category;
• DLT publication successes and failures;
• duplicate-event count;
• rebalance count and duration;
• handler transaction duration;
• oldest unprocessed event age;
• replay counts and outcomes.

Avoid labels such as raw event ID in metrics because unbounded cardinality can overwhelm the monitoring backend. Put event IDs in structured logs or traces, subject to privacy rules.

The most useful alert is often not “consumer is running.” It is “the oldest eligible business event has not completed within its service-level objective.”

Test Failure Windows, Not Only the Listener Method

A production-oriented test suite should cover:

1. a valid record produces one database effect;
2. the same event ID delivered twice produces one effect;
3. a transient exception succeeds on a later attempt;
4. an invalid event skips retry and reaches the DLT;
5. an exhausted transient failure reaches the DLT;
6. DLT headers preserve original coordinates;
7. a crash after database commit but before offset commit leads to a safe duplicate;
8. multiple instances racing on the same event ID respect the unique constraint;
9. a rebalance during processing does not create an extra effect;
10. replay through the approved path is idempotent;
11. DLT publication failure is visible and does not silently discard the source record;
12. large or malformed records follow the deserialization-error policy.

Use Testcontainers or an equivalent real broker and database for integration tests. Mocking KafkaTemplate cannot reproduce offset commits, partition assignment, serialization, retry seeks, transactions, or rebalances.

Also test deserialization failures. A listener never receives a record that the deserializer cannot construct, so error handling must be configured at the consumer boundary rather than only inside the method.

Anti-Patterns to Remove

Infinite retries

They block progress and turn permanent data errors into incidents with growing lag.

Catch, log, and return

The offset can advance even though the business effect failed.

A DLT with no owner

It becomes silent data loss with extra storage.

Offset-based business deduplication only

It fails when an event is republished or moved. Prefer a stable event ID, optionally retaining original coordinates for audit.

Database calls inside a Kafka transaction presented as global atomicity

It hides commit-order failure windows and makes recovery harder to reason about.

Replaying directly into production at full speed

It can overload the dependency that caused the original failure and makes a bad repair expensive.

Production Checklist

• Each event has a stable ID, key, type, and schema version.
• Duplicate processing is safe at the business-effect boundary.
• Transient and permanent exceptions are classified.
• Retries are bounded and backoff is intentional.
• Partition ordering requirements are documented.
• DLT records retain approved recovery context.
• DLT retention, ownership, alerting, and replay are defined.
• Database, Kafka, and external-side-effect transaction boundaries are explicit.
• Lag, event age, retries, duplicates, rebalances, and DLT depth are observable.
• Deserialization, crash windows, rebalances, and replay are integration-tested.
• The team has rehearsed a small production-safe replay.

References

• Spring Kafka: Handling Exceptions
• Spring Kafka: Non-Blocking Retries
• Apache Kafka: Producer Configuration
• Apache Kafka: Consumer Configuration

Resilience is not the claim that every record is processed exactly once. It is the ability to explain every failure window, preserve the record's business intent, prevent repeated effects, and recover through a procedure the team has already tested.