Idempotent Kafka Consumers with Spring Boot and PostgreSQL

An idempotent Kafka consumer produces the same committed business state whether a record is delivered once or several times. That property matters even when producers use Kafka idempotence or transactions, because a consumer can commit PostgreSQL and crash before its Kafka offset is committed.

On restart, Kafka delivers the record again. The duplicate is expected behavior, not a broker defect.

The reliable Spring Boot pattern is:

1. Every business event carries a stable event ID.
2. The consumer inserts that ID into a PostgreSQL deduplication table.
3. The deduplication insert and business update commit in the same database transaction.
4. The Kafka listener returns successfully only after that transaction commits.

Version note

This article was reviewed against Spring Boot 4.1, current Spring for Apache Kafka documentation, and PostgreSQL 18 documentation on August 10, 2026.

TL;DR

• Design for at-least-once delivery and make the database effect idempotent.
• Use one atomic INSERT ... ON CONFLICT DO NOTHING, not an exists check followed by an insert.
• Store the dedup marker and business change in the same PostgreSQL transaction.
• Prefer a producer-generated business event ID over topic-partition-offset when events may be republished.
• Do not treat a remote HTTP call as protected by the database transaction. Persist an outbound intent or use the provider's idempotency contract.

The failure window that creates duplicates

Consider an OrderPaid record that credits loyalty points:

consume Kafka record
  -> update PostgreSQL
  -> commit PostgreSQL
  -> commit Kafka offset

If the process stops after the database commit but before the offset commit, the group resumes from the old offset. Without idempotency, the same points are credited twice.

Kafka transactions provide exactly-once semantics for Kafka read-process-write pipelines when offsets and output records are committed in one Kafka transaction. They do not automatically make an independent PostgreSQL transaction or an external HTTP side effect part of that atomic boundary.

The practical target is effectively-once database state: delivery can repeat, but a stable event produces one committed effect for a particular logical consumer.

Define a stable event envelope

The producer should create the ID once and preserve it across retries:

public record EventEnvelope<T>(
    UUID eventId,
    String eventType,
    int eventVersion,
    Instant occurredAt,
    String aggregateId,
    T data
) {}

public record OrderPaid(
    UUID orderId,
    UUID customerId,
    BigDecimal amount,
    String currency
) {}

The event ID identifies the fact, not the send attempt. A producer retry, outbox relay retry, or manual republication of that same fact must keep the ID.

Kafka coordinates are an alternative deduplication key:

(consumer group, topic, partition, offset)

That key uniquely identifies one Kafka record. It does not identify the same business event after it is copied to another topic or republished at a new offset. Prefer the business event ID when the producer contract can supply one; retain Kafka coordinates as diagnostic metadata.

Create a deduplication table

Scope the primary key to a logical consumer. Two independent consumers may both need to process the same event exactly once.

create table processed_kafka_event (
    consumer_name    text        not null,
    event_id         uuid        not null,
    event_type       text        not null,
    event_version    integer     not null,
    payload_hash     varchar(64) not null,
    topic            text        not null,
    partition_id     integer     not null,
    record_offset    bigint      not null,
    processed_at     timestamptz not null default now(),
    primary key (consumer_name, event_id)
);

create index ix_processed_kafka_event_time
    on processed_kafka_event (processed_at);

payload_hash detects a serious producer error: the same event ID arriving with different content. Silently treating that case as an ordinary duplicate hides corruption.

Do not make event_id globally unique unless only one logical handler is ever allowed to process it. A projection, notification module, and fraud module can legitimately consume the same fact.

Claim the event atomically

An existence query followed by an insert has a race:

consumer A: event does not exist
consumer B: event does not exist
consumer A: apply side effect
consumer B: apply side effect

Let the PostgreSQL unique constraint serialize that decision:

@Repository
public class ProcessedEventRepository {

    private final JdbcTemplate jdbc;

    public ProcessedEventRepository(JdbcTemplate jdbc) {
        this.jdbc = jdbc;
    }

    public boolean claim(
        String consumerName,
        EventEnvelope<?> event,
        String payloadHash,
        ConsumerRecord<?, ?> record
    ) {
        int inserted = jdbc.update("""
            insert into processed_kafka_event (
                consumer_name,
                event_id,
                event_type,
                event_version,
                payload_hash,
                topic,
                partition_id,
                record_offset
            ) values (?, ?, ?, ?, ?, ?, ?, ?)
            on conflict (consumer_name, event_id) do nothing
            """,
            consumerName,
            event.eventId(),
            event.eventType(),
            event.eventVersion(),
            payloadHash,
            record.topic(),
            record.partition(),
            record.offset()
        );

        return inserted == 1;
    }

    public String findPayloadHash(String consumerName, UUID eventId) {
        return jdbc.queryForObject("""
            select payload_hash
              from processed_kafka_event
             where consumer_name = ? and event_id = ?
            """, String.class, consumerName, eventId);
    }
}

ON CONFLICT DO NOTHING is atomic under concurrency. One transaction inserts the key; another attempting the same key waits as necessary and then observes the conflict.

Commit the marker with the business effect

The transaction boundary belongs around both operations:

@Service
public class LoyaltyEventProcessor {

    private static final String CONSUMER = "loyalty-order-paid-v1";

    private final ProcessedEventRepository processedEvents;
    private final LoyaltyAccountRepository loyaltyAccounts;

    public LoyaltyEventProcessor(
        ProcessedEventRepository processedEvents,
        LoyaltyAccountRepository loyaltyAccounts
    ) {
        this.processedEvents = processedEvents;
        this.loyaltyAccounts = loyaltyAccounts;
    }

    @Transactional
    public ProcessingResult process(
        EventEnvelope<OrderPaid> event,
        String payloadHash,
        ConsumerRecord<?, ?> record
    ) {
        boolean firstDelivery = processedEvents.claim(
            CONSUMER,
            event,
            payloadHash,
            record
        );

        if (!firstDelivery) {
            String originalHash = processedEvents.findPayloadHash(
                CONSUMER,
                event.eventId()
            );

            if (!MessageDigest.isEqual(
                originalHash.getBytes(StandardCharsets.US_ASCII),
                payloadHash.getBytes(StandardCharsets.US_ASCII)
            )) {
                throw new EventIdentityCollision(event.eventId());
            }

            return ProcessingResult.DUPLICATE;
        }

        OrderPaid paid = event.data();
        loyaltyAccounts.creditForOrder(
            paid.customerId(),
            paid.orderId(),
            pointsFor(paid.amount())
        );

        return ProcessingResult.APPLIED;
    }
}

If creditForOrder fails, Spring rolls back the loyalty update and the processed-event insert. The next delivery can try again. If both commit and the offset commit is later lost, the next delivery finds the marker and returns without applying the credit.

The ordering inside the transaction is important. Insert the marker first, then apply the effect. The unique constraint becomes the concurrency gate.

Keep the listener thin

The Kafka listener validates the envelope, computes a canonical fingerprint, and calls the transactional service:

@Component
public class OrderPaidListener {

    private final LoyaltyEventProcessor processor;
    private final EventFingerprint fingerprint;

    public OrderPaidListener(
        LoyaltyEventProcessor processor,
        EventFingerprint fingerprint
    ) {
        this.processor = processor;
        this.fingerprint = fingerprint;
    }

    @KafkaListener(
        topics = "commerce.order-paid.v1",
        groupId = "loyalty-order-paid-v1"
    )
    public void on(ConsumerRecord<String, EventEnvelope<OrderPaid>> record) {
        EventEnvelope<OrderPaid> event = requireValid(record.value());
        String hash = fingerprint.sha256(event);

        processor.process(event, hash, record);
    }
}

The listener returns normally for a true duplicate so the container can advance the offset. It throws for a transient database failure so the configured error handler can retry or recover the record.

Use a deterministic serializer for the fingerprint. Hashing arbitrary JSON text is unsafe because whitespace and property order can change without changing the data. One option is a canonical JSON representation; another is a producer-supplied content hash covered by the event contract.

Configure offset behavior deliberately

Disable Kafka's periodic auto commit and let the Spring Kafka listener container manage offsets:

spring:
  kafka:
    consumer:
      enable-auto-commit: false
      properties:
        isolation.level: read_committed

read_committed prevents consumers from seeing aborted records when producers use Kafka transactions. It does not add PostgreSQL to the Kafka transaction.

For a record listener, the container commits an offset after successful listener processing according to its acknowledgment configuration. Do not acknowledge before the database transaction completes. Manual immediate acknowledgment is usually unnecessary for this pattern and can accidentally move the offset ahead of the durable effect.

When using batch listeners, decide whether one failed record should roll back the entire database batch. Record listeners make the per-event transaction and deduplication contract easier to reason about.

Do not put an irreversible remote call in the gap

This is not protected by the PostgreSQL transaction:

@Transactional
public void process(OrderPaid event) {
    claim(event.eventId());
    paymentProvider.capture(event.orderId());
    updateDatabase(event);
}

If the provider succeeds and PostgreSQL rolls back, Kafka redelivery calls the provider again. A JDBC rollback cannot undo an HTTP response.

Choose one of these strategies:

• pass a stable idempotency key to a provider that documents idempotent requests;
• store a durable outbound command in PostgreSQL and dispatch it separately;
• use a transactional outbox for a downstream Kafka message;
• add a reconciliation process when the remote API offers only status lookup.

The reliable outbound delivery guide covers this boundary in depth.

Retries and dead-letter topics do different jobs

Idempotency protects repeated successful effects. Retry and recovery policies decide what to do when processing cannot succeed.

Classify failures:

• Transient: connection timeout, temporary lock timeout, short dependency outage.
• Permanent data error: unsupported event version, invalid currency, missing required field.
• Code or schema defect: deserialization mismatch, constraint that the contract violates.

Use bounded retries for transient failures. Route permanent failures to a recoverable dead-letter path with the original topic, partition, offset, event ID, exception class, and deployment version. An unbounded retry on one partition can block every later record in that partition.

The Spring Kafka retry and recovery guide covers ordering, dead-letter records, and replay procedures.

Retention must match replay policy

Deleting processed-event rows too early re-enables old side effects. Before pruning, account for:

• Kafka topic retention;
• compacted-topic history;
• dead-letter retention;
• backup restoration windows;
• disaster recovery;
• manual replay and backfill policy;
• the maximum time an outbox relay can lag.

If the business permits replay after the dedup window, give the replay a new consumer identity or an explicit replay mode. Never let a cleanup job silently redefine the delivery guarantee.

At high volume, monitor table and index growth, autovacuum, and cleanup duration. Time-based partitioning can help retention, but PostgreSQL requires partitioned unique constraints to include the partition key. That affects the primary-key design, so do not add partitioning without modeling the uniqueness rule first.

Test the crash windows

A normal integration test proves only the happy path. Use a real PostgreSQL and Kafka test environment and cover:

Also test a rebalance and process termination, not just a thrown Java exception. The most important duplicate occurs after a successful database commit.

Observe idempotency as a feature

Track:

• applied and duplicate event counts by bounded event type;
• event identity collisions;
• transaction latency and rollback count;
• consumer lag and rebalance count;
• retry attempts and dead-letter records;
• oldest retained processed-event row and cleanup progress.

Do not use eventId as a metric label. Put it in structured logs or trace attributes so individual cases remain searchable without creating unbounded metric cardinality.

Common mistakes

Checking then inserting

The two queries race. Enforce uniqueness and use one atomic insert.

Committing the marker separately

If the marker commits and the business update fails, every retry is skipped. They must share one transaction.

Generating a new ID in the consumer

Every delivery receives a different ID, so no duplicate can be recognized. The ID belongs to the event fact and is created by the producer.

Using only Kafka offset as business identity

It works for repeat delivery of one record but not republication at another offset. Store both the stable event ID and Kafka coordinates.

Assuming Kafka exactly-once covers PostgreSQL

Kafka's guarantee has a defined resource boundary. External database and HTTP effects still require idempotency or durable intent.

Swallowing every exception

If the listener logs a database failure and returns normally, the offset can advance while the effect is missing. Return normally only for an already-committed duplicate or an explicitly recovered record.

Production checklist

• Does every event have a stable producer-generated ID?
• Is deduplication scoped to the correct logical consumer?
• Does PostgreSQL enforce the dedup key with a unique constraint?
• Are marker insert and business effect in the same transaction?
• Is an event-ID/payload mismatch detected?
• Does the listener return only after the database transaction completes?
• Are remote side effects protected separately?
• Are retries bounded and permanent failures recoverable?
• Does dedup retention cover every supported replay window?
• Have commit-before-offset crashes and concurrent duplicates been tested?

For the complete producer-outbox and consumer-dedup architecture, see Exactly-once Kafka processing with Spring Boot and PostgreSQL. The essential consumer rule remains simple: persist the event identity and its database effect together, then assume Kafka may deliver the record again.

Official references

• Apache Kafka documentation
• Spring for Apache Kafka: Exactly-once semantics
• Spring for Apache Kafka: Transactions
• Spring for Apache Kafka: Committing offsets
• Spring Framework: Transaction management
• PostgreSQL: INSERT and ON CONFLICT