Exactly-Once Kafka Processing with Spring Boot and PostgreSQL: Outbox and Idempotent Consumers

Exactly-once Kafka processing is easy to describe and surprisingly easy to overstate. Kafka can make a group of Kafka writes and consumed offsets atomic, but a business workflow often crosses a second system such as PostgreSQL. Once a database transaction enters the picture, Kafka alone cannot guarantee that every external side effect happens exactly once.

This article builds a more precise target: effectively-once business processing. Messages may be delivered again after a crash or rebalance, but committed business state changes only once. The design uses a transactional outbox on the producer side, at-least-once publication to Kafka, and an idempotent consumer transaction in PostgreSQL.

TL;DR Write business data and an outbox event in one PostgreSQL transaction. Publish outbox rows at least once and keep a stable event ID on every record. In each consumer group, insert that event ID and update business state in the same database transaction.

Start by Defining the Transaction Boundary

The phrase “exactly once” means different things depending on where the transaction begins and ends.

Kafka-only processing

Kafka transactions can atomically combine:

• records produced to one or more Kafka topics or partitions;
• offsets consumed from an input topic;
• visibility of committed records to consumers configured with read_committed.

This is a strong fit for a consume-transform-produce pipeline whose durable state stays inside Kafka.

Kafka plus PostgreSQL

A PostgreSQL commit and a Kafka commit belong to different resource managers. Without a distributed transaction protocol, there is always a failure window between them.

Consider a service that performs these steps:

1. inserts an order in PostgreSQL;
2. sends OrderCreated to Kafka;
3. returns success to the caller.

Two orderings are possible, and both have a dangerous window:

• Database first: the database commits, then the process crashes before Kafka accepts the event.
• Kafka first: Kafka accepts the event, then the database transaction rolls back.

Trying to hide this window behind a single @Transactional annotation does not make the two systems one atomic resource. For this boundary, the practical goal is to make retries safe and recovery deterministic.

The Recommended Architecture

The end-to-end flow looks like this:

HTTP request
    |
    v
Producer service
    |
    | one PostgreSQL transaction
    +--> business table
    +--> outbox_events
             |
             v
       outbox relay
             |
             | at-least-once publish
             v
           Kafka
             |
             v
Consumer service
    |
    | one PostgreSQL transaction
    +--> processed_messages
    +--> business side effect

Each layer has one clear responsibility:

The important design choice is deliberate: the relay is allowed to publish duplicates. The consumer is responsible for making those duplicates harmless.

Producer Side: Store the Event with the Business Change

An outbox event needs a stable identifier that survives every retry. Generate it when the business transaction is created, not inside the relay.

A practical PostgreSQL schema is:

CREATE TYPE outbox_status AS ENUM (
    'NEW',
    'IN_PROGRESS',
    'PUBLISHED',
    'FAILED'
);

CREATE TABLE outbox_events (
    id UUID PRIMARY KEY,
    aggregate_type VARCHAR(100) NOT NULL,
    aggregate_id VARCHAR(200) NOT NULL,
    event_type VARCHAR(150) NOT NULL,
    payload JSONB NOT NULL,
    status outbox_status NOT NULL DEFAULT 'NEW',
    attempts INTEGER NOT NULL DEFAULT 0,
    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    locked_at TIMESTAMPTZ,
    published_at TIMESTAMPTZ,
    last_error TEXT
);

CREATE INDEX idx_outbox_claim
    ON outbox_events (status, created_at)
    WHERE status IN ('NEW', 'IN_PROGRESS');

The producer writes the domain object and event in the same local transaction:

@Service
public class OrderService {

    private final OrderRepository orderRepository;
    private final OutboxEventRepository outboxRepository;
    private final ObjectMapper objectMapper;

    public OrderService(
            OrderRepository orderRepository,
            OutboxEventRepository outboxRepository,
            ObjectMapper objectMapper
    ) {
        this.orderRepository = orderRepository;
        this.outboxRepository = outboxRepository;
        this.objectMapper = objectMapper;
    }

    @Transactional
    public Order createOrder(CreateOrderCommand command) {
        Order order = orderRepository.save(
                Order.create(
                        command.customerId(),
                        command.totalAmount()
                )
        );

        UUID eventId = UUID.randomUUID();

        OrderCreated payload = new OrderCreated(
                eventId,
                order.getId(),
                order.getCustomerId(),
                order.getTotalAmount(),
                Instant.now()
        );

        OutboxEvent outboxEvent = OutboxEvent.newEvent(
                eventId,
                "Order",
                order.getId().toString(),
                "OrderCreated",
                writeJson(payload)
        );

        outboxRepository.save(outboxEvent);
        return order;
    }

    private String writeJson(Object value) {
        try {
            return objectMapper.writeValueAsString(value);
        } catch (JsonProcessingException exception) {
            throw new IllegalArgumentException(
                    "Could not serialize the outbox payload",
                    exception
            );
        }
    }
}

If serialization or the outbox insert fails, the order transaction rolls back. If the transaction commits, the event intent is durable even when Kafka is temporarily unavailable.

Why not call kafkaTemplate.send() in this method?

A direct send creates an ambiguous result around process failure. The database can commit while the send has not completed, or the broker can accept the record while the application never receives the acknowledgement.

The outbox changes the problem from “atomically commit two systems” to “reliably move a durable row from one system to another.” That second problem is much easier to recover.

Relay Side: Claim, Publish, and Finalize

A production relay must handle multiple application instances without publishing every row concurrently. PostgreSQL's FOR UPDATE SKIP LOCKED is useful for claiming independent batches.

The following query atomically claims a batch:

WITH candidates AS (
    SELECT id
    FROM outbox_events
    WHERE status = 'NEW'
       OR (
            status = 'IN_PROGRESS'
            AND locked_at < NOW() - INTERVAL '5 minutes'
       )
    ORDER BY created_at
    FOR UPDATE SKIP LOCKED
    LIMIT :batch_size
)
UPDATE outbox_events AS event
SET status = 'IN_PROGRESS',
    locked_at = NOW(),
    attempts = event.attempts + 1
FROM candidates
WHERE event.id = candidates.id
RETURNING event.*;

The stale-lock condition recovers rows claimed by an instance that crashed.

Keep the claiming transaction short. Do not hold database row locks while waiting on Kafka. A relay can follow this sequence:

1. claim a batch in one short PostgreSQL transaction;
2. publish each record and wait for Kafka's completion result;
3. mark a successful row as PUBLISHED in a new PostgreSQL transaction;
4. return a failed row to NEW, or move it to FAILED after a configured threshold.

A simplified publisher looks like this:

@Component
public class OutboxRelay {

    private static final String EVENT_ID_HEADER = "event-id";

    private final OutboxClaimService claimService;
    private final OutboxStateService stateService;
    private final KafkaTemplate<String, String> kafkaTemplate;

    public OutboxRelay(
            OutboxClaimService claimService,
            OutboxStateService stateService,
            KafkaTemplate<String, String> kafkaTemplate
    ) {
        this.claimService = claimService;
        this.stateService = stateService;
        this.kafkaTemplate = kafkaTemplate;
    }

    @Scheduled(fixedDelayString = "${outbox.poll-delay:1000}")
    public void publishBatch() {
        List<ClaimedOutboxEvent> events = claimService.claim(100);

        for (ClaimedOutboxEvent event : events) {
            publishOne(event);
        }
    }

    private void publishOne(ClaimedOutboxEvent event) {
        ProducerRecord<String, String> record = new ProducerRecord<>(
                topicFor(event.eventType()),
                event.aggregateId(),
                event.payload()
        );

        record.headers().add(
                EVENT_ID_HEADER,
                event.id().toString().getBytes(StandardCharsets.UTF_8)
        );

        try {
            kafkaTemplate.send(record).get(10, TimeUnit.SECONDS);
            stateService.markPublished(event.id());
        } catch (InterruptedException exception) {
            Thread.currentThread().interrupt();
            stateService.releaseForRetry(
                    event.id(),
                    "Interrupted while publishing"
            );
        } catch (ExecutionException | TimeoutException exception) {
            stateService.releaseForRetry(
                    event.id(),
                    exception.getMessage()
            );
        }
    }

    private String topicFor(String eventType) {
        return switch (eventType) {
            case "OrderCreated" -> "order-events";
            default -> throw new IllegalArgumentException(
                    "Unsupported event type: " + eventType
            );
        };
    }
}

Waiting for the returned future matters. Calling send() and immediately setting PUBLISHED can lose events because the send completes asynchronously.

The unavoidable duplicate window

Even this relay has a small failure window:

1. Kafka accepts the event.
2. The process crashes before PostgreSQL records PUBLISHED.
3. The stale claim is recovered.
4. The relay sends the same event again.

That is why the stable outbox id must travel with the message. The relay provides at-least-once publication, and the consumer turns that into effectively-once database state.

For high-volume systems, change-data capture with Debezium is another valid relay implementation. It removes application polling, but it does not remove the need for stable event IDs and idempotent consumers.

Consumer Side: Insert the Deduplication Marker First

A separate deduplication table should scope uniqueness to a consumer group. Two consumer groups may legitimately process the same event for different business purposes.

CREATE TABLE processed_messages (
    consumer_group_id VARCHAR(200) NOT NULL,
    event_id UUID NOT NULL,
    processed_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    PRIMARY KEY (consumer_group_id, event_id)
);

Avoid a check-then-insert sequence such as:

SELECT whether the event exists
then
INSERT the event

Two concurrent transactions can both observe “not found.” Use the database constraint as the concurrency control instead.

@Repository
public class ProcessedMessageStore {

    private final JdbcTemplate jdbcTemplate;

    public ProcessedMessageStore(JdbcTemplate jdbcTemplate) {
        this.jdbcTemplate = jdbcTemplate;
    }

    public boolean claim(String consumerGroupId, UUID eventId) {
        int inserted = jdbcTemplate.update(
                """
                INSERT INTO processed_messages (
                    consumer_group_id,
                    event_id
                )
                VALUES (?, ?)
                ON CONFLICT DO NOTHING
                """,
                consumerGroupId,
                eventId
        );

        return inserted == 1;
    }
}

The listener claims the event ID and changes business state in one PostgreSQL transaction:

@Service
public class OrderProjectionListener {

    private static final String CONSUMER_GROUP = "order-projection-v1";
    private static final String EVENT_ID_HEADER = "event-id";

    private final ProcessedMessageStore processedMessageStore;
    private final OrderProjectionRepository projectionRepository;
    private final ObjectMapper objectMapper;

    public OrderProjectionListener(
            ProcessedMessageStore processedMessageStore,
            OrderProjectionRepository projectionRepository,
            ObjectMapper objectMapper
    ) {
        this.processedMessageStore = processedMessageStore;
        this.projectionRepository = projectionRepository;
        this.objectMapper = objectMapper;
    }

    @KafkaListener(
            topics = "order-events",
            groupId = CONSUMER_GROUP
    )
    @Transactional
    public void onOrderEvent(ConsumerRecord<String, String> record) {
        UUID eventId = readEventId(record);
        OrderCreated event = readPayload(record.value());

        boolean firstAttempt = processedMessageStore.claim(
                CONSUMER_GROUP,
                eventId
        );

        if (!firstAttempt) {
            return;
        }

        projectionRepository.upsertCreatedOrder(
                event.orderId(),
                event.customerId(),
                event.totalAmount(),
                event.occurredAt()
        );
    }

    private UUID readEventId(ConsumerRecord<String, String> record) {
        Header header = record.headers().lastHeader(EVENT_ID_HEADER);

        if (header == null) {
            throw new IllegalArgumentException(
                    "Missing event-id header"
            );
        }

        return UUID.fromString(
                new String(header.value(), StandardCharsets.UTF_8)
        );
    }

    private OrderCreated readPayload(String json) {
        try {
            return objectMapper.readValue(json, OrderCreated.class);
        } catch (JsonProcessingException exception) {
            throw new IllegalArgumentException(
                    "Invalid OrderCreated payload",
                    exception
            );
        }
    }
}

There is intentionally no manual acknowledgement in this method.

The sequence is:

1. Spring invokes the listener.
2. PostgreSQL inserts the deduplication marker.
3. PostgreSQL applies the business update.
4. The database transaction commits.
5. After the listener returns successfully, the container advances the Kafka offset according to its acknowledgement mode.

If the database transaction fails, the exception escapes and the record can be retried. If the database commits but the process dies before the offset is committed, Kafka redelivers the event. The second delivery reaches ON CONFLICT DO NOTHING and returns without applying the business update again.

This is the core effectively-once mechanism.

Consumer Configuration

A record listener commonly starts with:

spring:
  kafka:
    bootstrap-servers: localhost:9092
    consumer:
      enable-auto-commit: false
      auto-offset-reset: earliest
    listener:
      ack-mode: record

Do not invent acknowledgement modes or assume that calling acknowledge() inside a database transaction makes the Kafka offset part of that transaction. The database and Kafka still commit independently.

Use a container error handler for retry and dead-letter behavior. The listener should throw on failures it cannot safely handle; logging an exception and returning normally tells the container that processing succeeded.

A useful error policy distinguishes:

• transient infrastructure failures that should be retried;
• malformed or permanently invalid events that should be sent to a dead-letter topic;
• duplicates, which are successful no-ops;
• business conflicts that require an explicit domain decision.

Where Kafka Transactions Still Help

The outbox and idempotent-consumer design solves a PostgreSQL boundary. Kafka transactions solve a different boundary.

Suppose a service:

1. consumes from payments;
2. transforms the record;
3. publishes to payment-results;
4. has no external database side effect.

In that Kafka-only pipeline, Spring Kafka can bind produced records and consumed offsets to one Kafka transaction. Spring Boot auto-configures a KafkaTransactionManager when a transaction ID prefix is provided:

spring:
  kafka:
    producer:
      transaction-id-prefix: payment-worker-${HOSTNAME:local}-
    consumer:
      properties:
        isolation.level: read_committed

The prefix must be unique for each application instance. Downstream consumers that must ignore aborted transactional records use read_committed.

Kafka transactions do not make a PostgreSQL commit atomic with a Kafka commit. Spring can synchronize multiple transaction managers and control commit order, but a second-resource commit can still fail after the first one succeeds. The Spring Kafka documentation therefore warns that database work in such a listener must remain idempotent.

Also avoid building new designs around ChainedKafkaTransactionManager; Spring Kafka marks it as deprecated.

Failure Matrix

A good design is easier to trust when every crash point has an expected recovery path.

This table is more useful than claiming a universal exactly-once guarantee. It explains what the system does when reality becomes inconvenient.

Testing the Failure Windows

A happy-path integration test is not enough. Verify the crash boundaries directly.

Producer transaction test

• Force the outbox insert to fail.
• Confirm that the order row also rolls back.
• Commit an order while Kafka is unavailable.
• Confirm that the outbox row remains and publishes after Kafka returns.

Relay duplicate test

• Publish an event successfully.
• Prevent the PUBLISHED update from committing.
• Allow the stale claim to be recovered.
• Confirm that Kafka receives the same event-id again.

Consumer idempotency test

• Send the same event ID twice.
• Confirm that the business table changes once.
• Confirm that processed_messages contains one row.

Consumer crash-window test

• Commit the PostgreSQL transaction.
• Stop the consumer before its offset advances.
• Restart it.
• Confirm that the redelivered event becomes a no-op.

Use real Kafka and PostgreSQL instances in integration tests. Testcontainers is a practical way to make those tests repeatable in CI without replacing the systems with mocks.

Operational Considerations

Keep the event ID immutable

The outbox row ID should become the event ID. Do not generate a new ID each time the relay retries, or the consumer cannot recognize duplicates.

Choose the message key deliberately

Use the aggregate ID as the Kafka key when events for one aggregate require partition ordering. All events for the same order then route to the same partition, assuming the partition count and partitioner remain compatible.

Clean up safely

The outbox and deduplication tables grow over time.

• Archive or delete PUBLISHED outbox rows after an operationally safe period.
• Retain deduplication records longer than the maximum interval in which an old event can be replayed.
• Do not base cleanup only on normal topic retention when operators can restore backups or replay archived topics.

Monitor the backlog

Useful metrics include:

• oldest NEW outbox event age;
• count of NEW, IN_PROGRESS, and FAILED events;
• publish attempts and failures;
• consumer lag;
• duplicate event count;
• dead-letter rate;
• database transaction latency.

An old outbox row is often more actionable than a generic “publisher is healthy” status.

External APIs need their own idempotency boundary

A consumer that calls a payment provider or email API cannot roll back that remote side effect with its PostgreSQL transaction.

Use one of these approaches:

• a provider-supported idempotency key based on the stable event ID;
• a second local outbox that records the outbound command;
• a state machine that records attempts and reconciles ambiguous responses.

The same rule applies recursively: every non-transactional boundary needs an explicit retry and deduplication strategy.

Troubleshooting

The relay keeps publishing the same event

Check whether:

• kafkaTemplate.send() completion is awaited;
• markPublished() runs in a real Spring transaction;
• self-invocation is bypassing a transactional proxy;
• stale IN_PROGRESS rows are recovered intentionally;
• the database update fails after Kafka succeeds.

Some duplicate publication is expected after a crash. The consumer must still be idempotent.

Duplicate events still change business state twice

Check whether:

• every retry preserves the original event ID;
• the primary key is (consumer_group_id, event_id);
• the marker insert and business update share one database transaction;
• code catches an exception and returns after a partial commit;
• another side effect occurs outside PostgreSQL, such as an external HTTP call.

Failed records disappear

Check whether the listener catches broad exceptions, logs them, and returns normally. Unless the error is deliberately converted into a successful no-op, let it escape so the configured error handler can retry or dead-letter the record.

read_committed does not stop duplicates

read_committed hides records from aborted Kafka transactions. It does not deduplicate:

• outbox relay retries;
• consumer redelivery after an offset-commit failure;
• repeated business commands that produce distinct event IDs.

Database idempotency remains necessary.

Conclusion

End-to-end exactly-once processing is not one configuration switch. It is a set of explicit guarantees at specific boundaries.

For a Spring Boot service that writes PostgreSQL data and publishes Kafka events:

• commit business state and the outbox row together;
• publish outbox events at least once;
• propagate one immutable event ID;
• claim the event ID and apply consumer state in one PostgreSQL transaction;
• treat Kafka transactions as a Kafka-only atomicity tool, not as a replacement for the outbox;
• test every crash window rather than relying on the happy path.

This design accepts the reality of redelivery and makes it harmless. That is the practical meaning of effectively-once processing.

Official References

• Spring Boot: Apache Kafka Support
• Spring for Apache Kafka: Transactions
• Spring for Apache Kafka: Transactions with Other Transaction Managers
• Apache Kafka: Producer Configuration
• Apache Kafka: Design and Exactly-Once Semantics