Eventual Consistency with Spring Boot, Kafka, and PostgreSQL: Convergence and Reconciliation

Eventual consistency is not a promise that “the data will probably match later.” It is a contract describing which copies may temporarily differ, how they converge, how long convergence should take, and what happens when they do not converge.

A typical system has several representations of one business fact:

Order service PostgreSQL row
  -> transactional outbox
  -> Kafka event
  -> inventory reservation
  -> billing projection
  -> customer-facing read model
  -> search index
  -> cache

Those representations cannot be committed through one ordinary local transaction. They may become visible at different times, receive duplicate events, or stop progressing after a consumer failure.

The design must therefore answer:

• Which representation is authoritative?
• Which states are allowed to be temporarily different?
• What is the maximum acceptable convergence delay?
• How does a consumer reject duplicates and stale events?
• What should a user see immediately after a write?
• How is permanent drift detected?
• Which discrepancies can be repaired automatically?
• Which discrepancies require a business decision?

This guide uses Spring Boot 4.1, Java 25, PostgreSQL, and Apache Kafka. It focuses on convergence, visibility, reconciliation, and repair. Outbox relays and workflow orchestration appear only where they establish the consistency boundary.

TL;DR Keep strong consistency inside one local transaction and use eventual consistency only across boundaries that cannot or should not share that transaction. Define a measurable convergence objective. Publish committed changes durably, consume at least once, and make projection updates idempotent and version-aware. Return a consistency token when users need read-your-writes behavior. Reconcile source and projection independently, and repair through an audited command rather than editing databases blindly.

Eventual Consistency Is a Choice, Not an Automatic Microservice Property

A microservice architecture does not require every operation to be eventually consistent.

Use a local PostgreSQL transaction when one service owns all data required by an invariant:

debit and credit inside one ledger database
order state and its outbox row
inventory quantity and reservation record
projection update and processed-event record

Use eventual consistency when independent ownership or availability requirements justify separate commits:

order confirmation
  -> inventory reservation
  -> invoice creation
  -> analytics update

The boundary should be explicit.

Do not move a rule out of a database transaction merely because Kafka is available.

Avoid an Oversimplified CAP Explanation

CAP concerns behavior during a network partition between distributed replicas or nodes. It does not mean every application must permanently choose either “consistency” or “availability” for every request.

A useful engineering question is narrower:

If part of the system cannot communicate,
which operations continue,
which operations stop,
and what temporary divergence is allowed?

For example:

• the order service may accept an order while analytics is unavailable;
• inventory reservation may fail closed when stock cannot be verified;
• a product catalog may serve slightly stale descriptions;
• a ledger transfer may reject the request rather than accept uncertain state.

Consistency policy belongs to each business operation.

Define a Convergence Contract

For every replicated or projected representation, document:

source of truth
projection owner
event or change source
ordering key
idempotency key
expected convergence time
maximum tolerated staleness
user-visible intermediate state
retry and dead-letter policy
reconciliation query
repair owner
retention required for replay

Example:

Projection: customer order history
Source: order-service PostgreSQL
Event: OrderConfirmed v1
Kafka key: orderId
Target: order-history PostgreSQL
Expected convergence: p99 under 30 seconds
Maximum stale window: 5 minutes
Read-your-writes: consistency token supported
Repair: replay one order snapshot
Owner: customer-experience team

“Eventually” becomes operationally meaningful only after it has a deadline.

Model Progress as Business State

Do not return a final-looking state while downstream work is still pending.

Weak model:

Order status = COMPLETED

even though inventory and payment are still processing.

A clearer model:

public enum OrderStatus {
    DRAFT,
    CONFIRMED,
    RESERVATION_PENDING,
    RESERVED,
    REJECTED,
    CANCELLED
}

The exact states depend on the domain. The important point is that temporary divergence is visible and understandable.

A user interface can then say:

Order accepted
Inventory confirmation pending

instead of showing success and silently changing it later.

Project Setup

Let Spring Boot manage compatible library versions.

<parent>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-parent</artifactId>
    <version>4.1.0</version>
    <relativePath />
</parent>

<properties>
    <java.version>25</java.version>
</properties>

<dependencies>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-webmvc</artifactId>
    </dependency>

    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-data-jpa</artifactId>
    </dependency>

    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-kafka</artifactId>
    </dependency>

    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-actuator</artifactId>
    </dependency>

    <dependency>
        <groupId>org.postgresql</groupId>
        <artifactId>postgresql</artifactId>
        <scope>runtime</scope>
    </dependency>

    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-test</artifactId>
        <scope>test</scope>
    </dependency>

    <dependency>
        <groupId>org.springframework.kafka</groupId>
        <artifactId>spring-kafka-test</artifactId>
        <scope>test</scope>
    </dependency>
</dependencies>

Use Flyway or Liquibase for schema changes.

spring:
  jpa:
    hibernate:
      ddl-auto: validate
    open-in-view: false

  kafka:
    consumer:
      enable-auto-commit: false
    listener:
      ack-mode: record

The application should not commit a Kafka offset after a projection update failed.

Establish the Durable Publication Boundary

A local change and its integration intent belong in one PostgreSQL transaction.

CREATE TABLE order_event_outbox (
    event_id UUID PRIMARY KEY,
    aggregate_id UUID NOT NULL,
    aggregate_version BIGINT NOT NULL,
    event_type VARCHAR(150) NOT NULL,
    schema_version INTEGER NOT NULL,
    payload JSONB NOT NULL,
    status VARCHAR(30) NOT NULL DEFAULT 'READY',
    attempts INTEGER NOT NULL DEFAULT 0,
    next_attempt_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    published_at TIMESTAMPTZ,

    UNIQUE (
        aggregate_id,
        aggregate_version,
        event_type
    )
);

@Service
public class ConfirmOrderService {

    private final OrderRepository orders;
    private final OrderOutboxWriter outbox;
    private final EntityManager entityManager;
    private final Clock clock;

    @Transactional
    public ConfirmOrderResult confirm(UUID orderId) {
        Order order = orders.findById(orderId)
                .orElseThrow(
                        () -> new OrderNotFoundException(
                                orderId
                        )
                );

        OrderConfirmed domainEvent =
                order.confirm(clock.instant());

        entityManager.flush();

        outbox.append(
                OrderConfirmedEvent.from(
                        domainEvent,
                        order.getVersion()
                )
        );

        return new ConfirmOrderResult(
                order.getId(),
                order.getStatus(),
                order.getVersion()
        );
    }
}

The relay publishes the outbox row later. It may publish the same event more than once if it crashes after Kafka accepts the record but before PostgreSQL records completion.

That duplicate window is expected. The consumer owns idempotency.

Make Projection Updates Idempotent and Version-Aware

Kafka records can be redelivered. A consumer rebalance, process crash, or transaction failure can cause the same event to appear again.

Use a processed-event table:

CREATE TABLE processed_events (
    consumer_name VARCHAR(150) NOT NULL,
    event_id UUID NOT NULL,
    processed_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),

    PRIMARY KEY (
        consumer_name,
        event_id
    )
);

Projection table:

CREATE TABLE order_history (
    order_id UUID PRIMARY KEY,
    customer_id UUID NOT NULL,
    status VARCHAR(40) NOT NULL,
    total_amount NUMERIC(19, 2) NOT NULL,
    currency VARCHAR(3) NOT NULL,
    source_version BIGINT NOT NULL,
    source_occurred_at TIMESTAMPTZ NOT NULL,
    projected_at TIMESTAMPTZ NOT NULL
);

Insert the processed-event marker atomically:

@Repository
public class ProcessedEventRepository {

    private final JdbcClient jdbc;

    public ProcessedEventRepository(
            JdbcClient jdbc
    ) {
        this.jdbc = jdbc;
    }

    public boolean tryInsert(
            String consumerName,
            UUID eventId
    ) {
        int inserted = jdbc.sql("""
                INSERT INTO processed_events (
                    consumer_name,
                    event_id
                )
                VALUES (
                    :consumer_name,
                    :event_id
                )
                ON CONFLICT DO NOTHING
                """)
                .param(
                        "consumer_name",
                        consumerName
                )
                .param("event_id", eventId)
                .update();

        return inserted == 1;
    }
}

Do not implement idempotency with:

if (!processed.exists(eventId)) {
    processed.save(eventId);
}

Two concurrent deliveries can both pass the existence check. Let the database unique constraint decide.

Apply the event in the same local transaction:

@Component
public class OrderHistoryListener {

    private static final String CONSUMER =
            "order-history-v1";

    private final ProcessedEventRepository processed;
    private final OrderHistoryRepository history;

    @KafkaListener(
        topics = "orders.events.v1",
        groupId = "order-history-v1"
    )
    @Transactional
    public void onOrderConfirmed(
            OrderConfirmedEvent event
    ) {
        if (!processed.tryInsert(
                CONSUMER,
                event.eventId()
        )) {
            return;
        }

        history.upsertIfNewer(event);
    }
}

Version-aware upsert:

@Repository
public class OrderHistoryRepository {

    private final JdbcClient jdbc;

    public int upsertIfNewer(
            OrderConfirmedEvent event
    ) {
        return jdbc.sql("""
                INSERT INTO order_history (
                    order_id,
                    customer_id,
                    status,
                    total_amount,
                    currency,
                    source_version,
                    source_occurred_at,
                    projected_at
                )
                VALUES (
                    :order_id,
                    :customer_id,
                    'CONFIRMED',
                    :total_amount,
                    :currency,
                    :source_version,
                    :source_occurred_at,
                    NOW()
                )
                ON CONFLICT (order_id)
                DO UPDATE SET
                    customer_id =
                        EXCLUDED.customer_id,
                    status =
                        EXCLUDED.status,
                    total_amount =
                        EXCLUDED.total_amount,
                    currency =
                        EXCLUDED.currency,
                    source_version =
                        EXCLUDED.source_version,
                    source_occurred_at =
                        EXCLUDED.source_occurred_at,
                    projected_at =
                        NOW()
                WHERE order_history.source_version
                      < EXCLUDED.source_version
                """)
                .param(
                        "order_id",
                        event.orderId()
                )
                .param(
                        "customer_id",
                        event.customerId()
                )
                .param(
                        "total_amount",
                        event.totalAmount()
                )
                .param(
                        "currency",
                        event.currency()
                )
                .param(
                        "source_version",
                        event.aggregateVersion()
                )
                .param(
                        "source_occurred_at",
                        event.occurredAt()
                )
                .update();
    }
}

The processed-event insert and projection update commit together. If either fails, the Kafka listener throws and the record remains eligible for redelivery.

The version guard protects the projection from an older event arriving after a newer one.

Event IDs and Aggregate Versions Solve Different Problems

Use both.

event_id
  -> identifies one published fact
  -> deduplicates redelivery

aggregate_version
  -> orders changes for one aggregate
  -> rejects stale state
  -> detects missing transitions

Do not use the order ID as the processed-event ID. One order can produce many legitimate events.

Do not use a timestamp as the only ordering mechanism. Clocks can differ, events can share a timestamp, and a replay can occur long after the original event.

Key Kafka Records by Aggregate

Publish all events for one aggregate with the same key:

ProducerRecord<String, OrderEvent> record =
        new ProducerRecord<>(
                "orders.events.v1",
                event.orderId().toString(),
                event
        );

Kafka preserves record order within one partition. It does not provide one global order across the topic.

The version remains necessary because:

• a historical record may be replayed;
• an operator may republish an old event;
• several topics can update one projection;
• a producer may use the wrong key;
• the projection may be rebuilt from snapshots and events;
• duplicate delivery remains possible.

State Events and Delta Events Need Different Gap Handling

A state event contains enough information to replace the current projection:

{
  "orderId": "6f116a19-55d0-499e-a243-5b4297797186",
  "aggregateVersion": 14,
  "status": "CONFIRMED",
  "totalAmount": 129.90,
  "currency": "USD"
}

If version 14 arrives before version 13, a state projection can often accept 14 and ignore 13 later.

A delta event describes only a change:

{
  "orderId": "6f116a19-55d0-499e-a243-5b4297797186",
  "aggregateVersion": 14,
  "amountAdded": 20.00
}

Applying version 14 without version 13 may produce the wrong result.

For delta events:

incoming version == current version + 1
  -> apply

incoming version <= current version
  -> duplicate or stale; ignore

incoming version > current version + 1
  -> gap; pause or repair before continuing

Choose the event style according to replay, payload size, privacy, and consistency requirements.

Deletion Needs a Tombstone

Deleting the source row does not automatically remove projections.

Publish a versioned tombstone:

public record OrderDeletedEvent(
        UUID eventId,
        UUID orderId,
        long aggregateVersion,
        Instant occurredAt
) {}

The projection records the deletion version before removing or masking the visible row.

A retained tombstone prevents an old replayed event from recreating deleted data.

For regulated deletion, define how the deletion propagates to:

PostgreSQL projections
Redis
search indexes
object storage
analytics
Kafka compacted topics
backups and retention workflows

Eventual deletion needs its own completion and audit model.

Retries Do Not Guarantee Convergence

A retry policy should distinguish failure categories.

A dead-letter topic is not a successful end state. It is a durable record that normal convergence stopped.

Every DLT needs:

• an owner;
• an alert;
• a retention policy;
• a diagnosis workflow;
• a replay procedure;
• an audit trail.

Do not acknowledge a failed projection update after merely logging the exception.

Read-Your-Writes Is Stronger Than Eventual Reads

A user often expects to see a change immediately after submitting it.

Consider:

POST /orders/{id}/confirm
  -> order database commits version 12
  -> response succeeds
  -> GET /order-history/{id}
  -> projection still has version 11

The system may be working correctly, but the user sees stale data.

A read-your-writes strategy can return a consistency token:

public record ConfirmOrderResult(
        UUID orderId,
        long sourceVersion
) {}

Response:

HTTP/1.1 200 OK
X-Consistency-Version: 12

The projection endpoint accepts the minimum version:

GET /order-history/6f116a19-55d0-499e-a243-5b4297797186
X-Min-Source-Version: 12

Controller:

@RestController
@RequestMapping("/order-history")
public class OrderHistoryController {

    private final OrderHistoryQuery history;

    @GetMapping("/{orderId}")
    public ResponseEntity<?> find(
            @PathVariable UUID orderId,
            @RequestHeader(
                name = "X-Min-Source-Version",
                required = false
            )
            Long minimumVersion
    ) {
        OrderHistoryView view =
                history.find(orderId);

        if (minimumVersion != null
                && view.sourceVersion()
                        < minimumVersion) {
            return ResponseEntity
                    .status(
                            HttpStatus.ACCEPTED
                    )
                    .header(
                            "Retry-After",
                            "1"
                    )
                    .body(
                            new ProjectionPending(
                                    orderId,
                                    minimumVersion,
                                    view.sourceVersion()
                            )
                    );
        }

        return ResponseEntity.ok(view);
    }
}

Possible product behaviors include:

• return 202 Accepted while the projection catches up;
• wait briefly with a bounded deadline;
• route the read to the source service;
• show optimistic client state;
• return the write response as the immediate view.

Do not block indefinitely waiting for Kafka.

A Consistency Token Needs a Defined Scope

A per-order aggregate version works for one resource.

It does not prove freshness for:

an entire customer dashboard
a query spanning many orders
a global sales total
a projection built from several topics

Broader views may need:

• a source event position;
• one token per partition;
• a projection checkpoint;
• a workflow completion ID;
• a domain-specific readiness marker.

Do not advertise a token as globally monotonic unless the system actually provides a global ordering mechanism.

Reconciliation Is an Independent Correctness Check

Retries answer:

Did this delivery eventually succeed?

Reconciliation answers:

Does the target state now agree with the authoritative state?

These are different controls.

A bug can consume every Kafka record successfully and still build the wrong projection. Consumer lag can be zero while data is incorrect.

Define Reconciliation Records

Compare a stable source representation with the projection.

public record OrderConsistencyRecord(
        UUID orderId,
        long sourceVersion,
        String status,
        BigDecimal totalAmount,
        String currency,
        boolean deleted
) {}

A deterministic fingerprint can reduce network and comparison cost:

public record OrderConsistencyFingerprint(
        UUID orderId,
        long sourceVersion,
        String sha256
) {}

The hash must be calculated from a canonical representation:

stable field order
stable decimal representation
stable time zone
explicit null encoding
explicit schema version
no volatile projected_at field

A changed JSON property order must not create a false discrepancy.

Do Not Compare Only Counts

These two systems can have the same row count and different data.

Source:
A = 100
B = 200

Projection:
A = 200
B = 100

Useful checks include:

• missing target rows;
• extra target rows;
• source and target version;
• deterministic content hash;
• deletion tombstones;
• domain aggregates such as currency totals;
• records older than the permitted convergence window;
• records recently repaired;
• version gaps.

Counts are a coarse smoke signal, not proof of convergence.

Respect the Normal Convergence Window

A reconciliation job should not flag every in-flight event as corruption.

If the objective allows five minutes of staleness:

source changed 30 seconds ago
target not updated yet
-> pending, not drift

source changed 20 minutes ago
target still behind
-> discrepancy

Use a cutoff:

Instant cutoff = clock.instant()
        .minus(maximumConvergenceDelay);

Compare only source versions that should already have converged, or classify younger rows separately as PENDING.

Preserve Service Ownership

A normal application service should not join another service's private tables directly.

A reconciliation design can use:

• a source-owned snapshot API;
• a versioned export file;
• a CDC snapshot topic;
• a read-only replica exposed through a governed data contract;
• a privileged platform reconciliation job with audited access.

The source contract should be stable and paginated.

public interface AuthoritativeOrderSnapshot {

    Page<OrderConsistencyRecord> fetch(
            OrderCursor cursor,
            Instant changedBefore,
            int pageSize
    );
}

The projection side exposes its own records:

public interface OrderHistorySnapshot {

    Map<UUID, OrderConsistencyRecord> fetchByIds(
            Collection<UUID> orderIds
    );
}

This keeps reconciliation from importing another bounded context's JPA repositories and entities.

Persist Reconciliation Runs

CREATE TABLE reconciliation_runs (
    run_id UUID PRIMARY KEY,
    projection_name VARCHAR(150) NOT NULL,
    cutoff_at TIMESTAMPTZ NOT NULL,
    status VARCHAR(30) NOT NULL,
    scanned_count BIGINT NOT NULL DEFAULT 0,
    discrepancy_count BIGINT NOT NULL DEFAULT 0,
    started_at TIMESTAMPTZ NOT NULL,
    completed_at TIMESTAMPTZ,
    last_cursor VARCHAR(500),
    error_message VARCHAR(1000)
);

CREATE TABLE reconciliation_discrepancies (
    run_id UUID NOT NULL,
    resource_id UUID NOT NULL,
    discrepancy_type VARCHAR(50) NOT NULL,
    source_version BIGINT,
    target_version BIGINT,
    source_hash VARCHAR(64),
    target_hash VARCHAR(64),
    detected_at TIMESTAMPTZ NOT NULL,
    repair_status VARCHAR(30) NOT NULL,
    repair_id UUID,

    PRIMARY KEY (
        run_id,
        resource_id,
        discrepancy_type
    )
);

A durable cursor lets a large scan resume instead of restarting from the beginning.

Reconcile in Bounded Pages

@Component
public class OrderHistoryReconciler {

    private final AuthoritativeOrderSnapshot source;
    private final OrderHistorySnapshot target;
    private final ReconciliationRepository runs;
    private final Clock clock;
    private final Duration convergenceDelay;

    @Scheduled(
        cron = "${reconciliation.order-history.cron:0 */15 * * * *}"
    )
    public void reconcile() {
        UUID runId = UUID.randomUUID();
        Instant cutoff = clock.instant()
                .minus(convergenceDelay);

        runs.start(runId, cutoff);

        OrderCursor cursor = OrderCursor.start();

        try {
            while (true) {
                Page<OrderConsistencyRecord> page =
                        source.fetch(
                                cursor,
                                cutoff,
                                500
                        );

                if (page.items().isEmpty()) {
                    break;
                }

                Map<UUID, OrderConsistencyRecord>
                        projected =
                        target.fetchByIds(
                                page.items().stream()
                                        .map(
                                            OrderConsistencyRecord
                                                    ::orderId
                                        )
                                        .toList()
                        );

                for (OrderConsistencyRecord expected
                        : page.items()) {
                    compareAndRecord(
                            runId,
                            expected,
                            projected.get(
                                    expected.orderId()
                            )
                    );
                }

                cursor = page.nextCursor();
                runs.recordProgress(
                        runId,
                        cursor,
                        page.items().size()
                );

                if (!page.hasNext()) {
                    break;
                }
            }

            runs.complete(runId);
        } catch (RuntimeException exception) {
            runs.fail(
                    runId,
                    sanitize(exception)
            );
            throw exception;
        }
    }
}

Bound:

• page size;
• execution duration;
• concurrent workers;
• source and target query rate;
• memory use;
• repair rate.

A reconciliation job should not cause the outage it is intended to detect.

Classify Discrepancies

Useful categories:

public enum DiscrepancyType {
    MISSING_TARGET,
    EXTRA_TARGET,
    STALE_VERSION,
    CONTENT_MISMATCH,
    DELETE_NOT_APPLIED,
    VERSION_GAP
}

The classification determines repair policy.

Examples:

A same-version content mismatch is especially important. Replaying the same broken transformation may reproduce the same wrong result.

Repair Through a Command, Not a Manual SQL Edit

A repair request should be explicit and idempotent.

public record RebuildOrderHistory(
        UUID repairId,
        UUID orderId,
        long authoritativeVersion,
        OrderConsistencyRecord snapshot,
        String reason,
        Instant requestedAt
) {}

The projection applies it only when it is not older than current state:

INSERT INTO order_history (
    order_id,
    customer_id,
    status,
    total_amount,
    currency,
    source_version,
    source_occurred_at,
    projected_at
)
VALUES (
    :order_id,
    :customer_id,
    :status,
    :total_amount,
    :currency,
    :source_version,
    :source_occurred_at,
    NOW()
)
ON CONFLICT (order_id)
DO UPDATE SET
    customer_id = EXCLUDED.customer_id,
    status = EXCLUDED.status,
    total_amount = EXCLUDED.total_amount,
    currency = EXCLUDED.currency,
    source_version = EXCLUDED.source_version,
    source_occurred_at =
        EXCLUDED.source_occurred_at,
    projected_at = NOW()
WHERE order_history.source_version
      <= EXCLUDED.source_version;

Record:

who or what requested repair
source version
target version
reason
repair command ID
result
timestamp
trace ID

Manual SQL may be necessary during an incident, but it should be an exceptional, reviewed, and audited operation.

Automatic Repair Has a Narrow Safe Zone

Automatic repair is usually appropriate for a derived representation that can be rebuilt from an authoritative source:

search index
read model
cache
analytics projection
denormalized query table

Automatic repair is risky for:

ledger entries
payment state
inventory ownership
legal records
data owned by another bounded context
irreversible external side effects

A compensating transaction is a new business action, not deletion of history.

For example:

PaymentCaptured
then order cancelled
-> issue RefundRequested

not:
-> delete PaymentCaptured

Replay and Rebuild Are Different Operations

Replay reprocesses historical events.

Rebuild replaces a target from an authoritative snapshot.

Replay works when:

• the topic retains the complete required history;
• every historical schema is readable;
• transformations are deterministic;
• event order and gaps are understood;
• side effects are disabled or idempotent.

Rebuild works when:

• current authoritative state is sufficient;
• old events are unavailable or incompatible;
• the projection does not require full history;
• a versioned snapshot contract exists.

For a large projection, rebuild into a new table or schema:

order_history_v2

Validate it, catch it up with live changes, then switch readers. Mutating the only live projection in place can make rollback difficult.

Kafka Retention Is Part of the Recovery Contract

A replay cannot restore records that Kafka retention has already removed.

Document:

maximum outage duration
topic retention
schema retention
consumer offset retention
snapshot frequency
time required to rebuild
storage required for replay

A 30-day topic does not support a 90-day historical rebuild unless another archive or snapshot exists.

Observe the Convergence Pipeline

A healthy broker does not prove that the business state converged.

Measure each boundary.

Publication

outbox rows by status
oldest READY row age
relay claim and publish rate
Kafka send failures
outbox retry count

Transport and consumption

consumer lag by group and partition
record age when consumed
deserialization failures
listener retry count
DLT count
rebalance count

Projection

projection apply latency
stale event rejection count
duplicate event count
version gap count
last successful checkpoint
database update failure count

Reconciliation and repair

records scanned
discrepancies by category
oldest unresolved discrepancy
repair requested, succeeded, and failed
same-version content mismatch
rebuild progress

Use bounded metric labels:

projection
event_type
outcome
discrepancy_type
repair_type

Do not use order IDs, customer IDs, or event IDs as metric labels.

Consumer Lag Is Necessary but Insufficient

Lag can answer:

How many records are waiting?

It cannot answer:

Were records transformed correctly?
Was one record skipped by a bug?
Did the target database contain an old manual edit?
Did every delete reach the projection?

Use lag for timeliness and reconciliation for correctness.

A projection with zero lag and 10,000 wrong rows is not healthy.

Measure Time Carefully

Useful timestamps include:

source occurred_at
outbox created_at
Kafka produced_at
consumer received_at
projection applied_at
reconciliation detected_at
repair completed_at

They help decompose delay:

publication delay
broker and queue delay
consumer processing delay
repair delay

Wall-clock comparisons are approximate when hosts have clock skew. Use synchronized clocks, but use aggregate versions and Kafka offsets for ordering rather than timestamps alone.

Health Endpoints Should Not Perform Deep Reconciliation

Do not query several databases, Kafka offsets, and external services inside every Kubernetes health probe.

Instead:

1. run freshness checks asynchronously;
2. cache a small status snapshot;
3. let the health indicator read that local snapshot.

public record ProjectionStatusSnapshot(
        boolean withinObjective,
        long lagRecords,
        Duration oldestPendingAge,
        Instant checkedAt
) {}

@Component
public class OrderHistoryReadiness
        implements HealthIndicator {

    private final ProjectionStatusMonitor monitor;

    @Override
    public Health health() {
        ProjectionStatusSnapshot snapshot =
                monitor.current();

        if (!snapshot.withinObjective()) {
            return Health
                    .status("OUT_OF_SERVICE")
                    .withDetail(
                            "lagRecords",
                            snapshot.lagRecords()
                    )
                    .withDetail(
                            "oldestPendingAge",
                            snapshot.oldestPendingAge()
                                    .toString()
                    )
                    .withDetail(
                            "checkedAt",
                            snapshot.checkedAt()
                    )
                    .build();
        }

        return Health.up()
                .withDetail(
                        "checkedAt",
                        snapshot.checkedAt()
                )
                .build();
    }
}

Use this as readiness only when an excessively stale projection means the instance should stop receiving traffic.

Do not make liveness depend on Kafka lag or PostgreSQL availability. Restarting the process repeatedly does not repair a slow consumer and can worsen a rebalance storm.

For projections that may serve stale data under a documented policy, alert instead of removing every replica from service.

Correlate Events Without Treating Traces as Proof

Include stable identifiers in structured logs:

event.id
aggregate.id
aggregate.version
topic
partition
offset
consumer.group
projection
repair.id
traceId

Distributed tracing helps locate processing delay and failures, but sampled traces cannot prove that every event was processed.

Durable event IDs, offsets, versions, inbox rows, and reconciliation records remain the correctness evidence.

User Experience Patterns

Pending resource

Return a workflow or projection status:

{
  "orderId": "6f116a19-55d0-499e-a243-5b4297797186",
  "status": "RESERVATION_PENDING",
  "lastConfirmedVersion": 12
}

Optimistic UI

Show the accepted change locally while the projection catches up, then reconcile with the server response.

This is suitable when rejection is rare and reversible. It is dangerous for balances, stock ownership, or irreversible actions.

Source fallback

For a short period after a write, read directly from the authoritative service.

This improves freshness but adds coupling and load. Bound the fallback duration and avoid creating a permanent synchronous dependency.

Bounded wait

Wait for the projection to reach the consistency token for a short deadline:

wait up to 500 ms
if caught up -> return 200
otherwise -> return 202

A bounded wait can improve common-case UX without hiding an outage behind an indefinite request.

Failure and Compensation

Not every downstream failure is a data inconsistency.

Example:

Order confirmed
Inventory rejects reservation because stock is unavailable

If rejection is a valid business outcome, the system should record it:

Order -> REJECTED or CANCELLED

It should not keep retrying forever until inventory happens to become available.

Distinguish:

• technical failure: operation did not execute; retry may be appropriate;
• business rejection: operation executed and returned a valid negative decision;
• ambiguous outcome: caller cannot prove whether the operation completed;
• data drift: durable states disagree beyond the convergence objective.

Each requires a different response.

Test the Failure Windows

Use PostgreSQL and Kafka Testcontainers for integration tests.

Local transaction atomicity

• fail the outbox insert;
• verify the order update rolls back;
• fail the order update;
• verify no outbox row exists.

Duplicate delivery

• send the same event_id twice;
• verify one processed-event row;
• verify one projection result.

Out-of-order state event

• apply aggregate version 12;
• then apply version 11;
• verify version 12 remains.

Delta-event gap

• apply version 10;
• send version 12 without 11;
• verify the consumer records a gap and does not apply an unsafe delta.

Crash after projection commit

• commit projection and inbox state;
• stop before offset progress completes;
• restart;
• verify redelivery is harmless.

Read-your-writes

• write source version 12;
• leave projection at version 11;
• request minimum version 12;
• verify the API returns pending rather than stale success;
• advance the projection;
• verify the same request returns the updated view.

Normal convergence window

• create a source change younger than the allowed delay;
• verify reconciliation classifies it as pending, not drift.

Missing projection

• create an old authoritative source record;
• omit it from the projection;
• verify a MISSING_TARGET discrepancy.

Same-version mismatch

• store source and target with the same version but different content;
• verify reconciliation does not treat the target as current merely because versions match.

Repair idempotency

• send the same repair_id twice;
• verify one durable result;
• send an older authoritative version;
• verify it cannot overwrite newer state.

Replay

• rebuild a clean projection from retained records;
• verify historical schemas remain readable;
• verify external side effects are disabled or idempotent.

Common Mistakes

“Eventual consistency is inevitable in every microservice”

No. Keep invariants in one local transaction when ownership permits it.

“Kafka delivered the event, so the system is consistent”

Delivery does not prove correct transformation or target persistence.

“Zero consumer lag proves correctness”

Lag measures backlog, not data equivalence.

“Retry every failure until it succeeds”

Business rejection and poison events do not become correct through infinite retries.

“Use timestamps to resolve every conflict”

Timestamps are not a reliable total order. Prefer source-owned revisions or versions.

“The latest event always wins”

Only when “latest” is defined by an authoritative ordering rule and the event contains safe replacement state.

“A health indicator should compare every database”

Deep comparisons inside probes create expensive and unstable health checks. Run reconciliation separately.

“The reconciliation service may edit every database”

Repair should respect data ownership and use explicit, audited commands.

“A DLT means the event was handled”

A DLT means normal processing stopped and requires ownership.

“A compensating transaction deletes the original event”

Compensation is a new business action. Preserve history.

Troubleshooting

Projection remains behind

Check:

• outbox oldest-row age;
• relay failures;
• Kafka topic and key;
• consumer group lag;
• listener retry state;
• DLT;
• target database latency;
• version-gap handling;
• deployment version.

Lag is zero, but records are missing

Check:

• source-to-outbox atomicity;
• topic retention;
• consumer filters;
• incorrect event keys;
• projection transaction rollback;
• manual offset changes;
• reconciliation results;
• application bugs that acknowledged without updating state.

Older state overwrites newer state

Check:

• aggregate version in the event;
• version-aware UPDATE predicate;
• all producers using the same key;
• replay tooling;
• repair commands;
• timestamp-based conflict logic.

Users see stale data immediately after a write

Add or inspect:

• source version in the write response;
• minimum-version request token;
• bounded wait behavior;
• source fallback;
• optimistic UI state;
• projection lag objective.

Reconciliation produces many false positives

Check:

• convergence cutoff;
• canonical serialization;
• decimal scale;
• time-zone normalization;
• deleted-record handling;
• snapshot consistency;
• pagination boundaries;
• changes occurring during the scan.

Repairs keep recreating the same mismatch

The projection transformation or schema may be wrong. Stop automatic repair and investigate the consumer logic before replaying again.

Review Checklist

Before production:

• Is the authoritative representation documented?
• Which rules stay strongly consistent?
• What is the convergence objective?
• Are pending states visible to users?
• Is publication durable?
• Are events keyed and versioned?
• Are consumers idempotent?
• Can stale events overwrite newer state?
• Are delta-event gaps detected?
• Are deletions represented by tombstones?
• Is read-your-writes required?
• What is the consistency token's scope?
• Are lag and data correctness measured separately?
• Does reconciliation ignore the normal convergence window?
• Are discrepancies persisted and classified?
• Which repairs are safe to automate?
• Are repair commands idempotent and audited?
• Does retention support the stated replay objective?
• Are health probes lightweight?
• Have duplicate, gap, crash, drift, and repair tests passed?

Conclusion

Eventual consistency is reliable only when convergence is designed as an observable and repairable process.

For Spring Boot, Kafka, and PostgreSQL:

• keep local invariants inside one PostgreSQL transaction;
• define a source of truth and convergence objective for every projection;
• publish committed change intent durably;
• process Kafka records at least once with an inbox or equivalent idempotency boundary;
• use source-owned versions to reject stale updates;
• distinguish state events from delta events;
• represent deletion explicitly;
• provide a scoped consistency token when users need read-your-writes behavior;
• monitor delay and correctness separately;
• reconcile source and projection after the normal propagation window;
• repair derived state through idempotent, audited commands;
• keep compensation as a new business action;
• test every durable failure window.

The goal is not to eliminate all temporary differences. The goal is to make every allowed difference bounded, visible, convergent, and recoverable.

Official References

• Spring Boot project
• Spring Boot Apache Kafka support
• Spring Kafka transactions
• Spring Kafka filtering and idempotent receivers
• Spring Boot Actuator health
• Apache Kafka design
• Apache Kafka monitoring
• PostgreSQL transaction isolation
• PostgreSQL transactions