Reliable External API Delivery with Spring Boot, PostgreSQL, and Kafka

External APIs fail in ways that local method calls do not. A provider can reject a request, rate-limit it, accept the operation and lose the response, or remain unavailable for hours. Your service can also crash after committing its own database transaction but before sending the request.

A reliable outbound integration therefore needs more than @Async, a retry annotation, or a Kafka topic. It needs a durable state machine whose failure windows are explicit.

This guide implements that state machine with Spring Boot 4.1, Java 25, PostgreSQL, RestClient, and optional Kafka coordination.

TL;DR Store the outbound intent in the same PostgreSQL transaction as the business change. Claim rows with FOR UPDATE SKIP LOCKED, perform HTTP outside the database transaction, and update the result in a separate transaction. Retry only outcomes that the provider contract makes safe, and use an idempotency key because a timeout can occur after the provider has already completed the operation.

What This Pattern Can Guarantee

A durable outbound gateway can guarantee that:

a committed business operation does not lose its outbound intent;
pending work survives application restarts;
several dispatcher replicas do not intentionally claim the same row at once;
retry timing and failure state are persisted;
operators can inspect and replay failed work.

It cannot independently guarantee that an external side effect occurs exactly once.

The external provider is outside your database transaction. After an HTTP request leaves your process, these events can happen:

Provider completes the request
Network drops the response
Client times out
Client retries
Provider receives a duplicate

A safe design needs at least one provider capability:

an idempotency key;
a request-status lookup API;
a client-supplied resource identifier;
a naturally idempotent operation;
a reconciliation process.

Without one of these, some failures are delivery unknown, not safely retryable.

The Core State Machine

public enum OutboundStatus {
    READY,
    IN_PROGRESS,
    RETRY_WAIT,
    SUCCEEDED,
    PERMANENT_FAILURE,
    DEAD,
    DELIVERY_UNKNOWN,
    CANCELLED
}

State	Meaning
`READY`	Eligible for first delivery
`IN_PROGRESS`	Claimed by one dispatcher
`RETRY_WAIT`	Retryable failure; waiting for `next_attempt_at`
`SUCCEEDED`	Provider returned a confirmed successful result
`PERMANENT_FAILURE`	Request is invalid or rejected permanently
`DEAD`	Retry budget exhausted
`DELIVERY_UNKNOWN`	Provider may have completed the operation, but the client cannot prove it
`CANCELLED`	Business intent was cancelled before delivery

DELIVERY_UNKNOWN is important. Treating every timeout as an ordinary retry can duplicate payments, shipments, emails, or account operations.

Database Schema

CREATE TABLE outbound_requests (
    id UUID PRIMARY KEY,
    business_key VARCHAR(200) NOT NULL,
    destination VARCHAR(100) NOT NULL,
    operation VARCHAR(100) NOT NULL,
    payload JSONB NOT NULL,

    status VARCHAR(40) NOT NULL,
    attempt_count INTEGER NOT NULL DEFAULT 0,
    max_attempts INTEGER NOT NULL DEFAULT 8,
    next_attempt_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),

    claim_token UUID,
    claimed_by VARCHAR(200),
    lease_until TIMESTAMPTZ,

    provider_request_id VARCHAR(250),
    last_http_status INTEGER,
    last_error_code VARCHAR(100),
    last_error_message VARCHAR(1000),

    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    succeeded_at TIMESTAMPTZ,

    version BIGINT NOT NULL DEFAULT 0,

    CONSTRAINT uk_outbound_business_operation
        UNIQUE (destination, operation, business_key)
);

CREATE INDEX idx_outbound_dispatch
    ON outbound_requests (
        next_attempt_at,
        created_at
    )
    WHERE status IN (
        'READY',
        'RETRY_WAIT',
        'IN_PROGRESS'
    );

The unique key prevents two application retries from scheduling the same business operation twice.

Do not rely on this unsafe sequence:

if (!repository.existsByBusinessKey(key)) {
    repository.save(request);
}

Two transactions can both observe that the row does not exist. Use a database unique constraint and an atomic insert.

Map JSON with Hibernate

@Entity
@Table(name = "outbound_requests")
public class OutboundRequest {

    @Id
    private UUID id;

    @Column(nullable = false, updatable = false)
    private String businessKey;

    @Column(nullable = false, updatable = false)
    private String destination;

    @Column(nullable = false, updatable = false)
    private String operation;

    @JdbcTypeCode(SqlTypes.JSON)
    @Column(columnDefinition = "jsonb", nullable = false)
    private JsonNode payload;

    @Enumerated(EnumType.STRING)
    @Column(nullable = false)
    private OutboundStatus status;

    @Column(nullable = false)
    private int attemptCount;

    @Column(nullable = false)
    private int maxAttempts;

    @Column(nullable = false)
    private Instant nextAttemptAt;

    private UUID claimToken;
    private String claimedBy;
    private Instant leaseUntil;

    private String providerRequestId;
    private Integer lastHttpStatus;
    private String lastErrorCode;
    private String lastErrorMessage;

    @Column(nullable = false)
    private Instant createdAt;

    @Column(nullable = false)
    private Instant updatedAt;

    private Instant succeededAt;

    @Version
    private long version;

    protected OutboundRequest() {}
}

Do not create a new ObjectMapper inside an entity constructor. Serialization configuration belongs in application infrastructure, and entity construction should not hide a possible serialization failure.

Store Intent in the Business Transaction

@Service
public class OrderApplicationService {

    private final OrderRepository orders;
    private final OutboundRequestWriter outbound;
    private final ObjectMapper objectMapper;

    public OrderApplicationService(
            OrderRepository orders,
            OutboundRequestWriter outbound,
            ObjectMapper objectMapper
    ) {
        this.orders = orders;
        this.outbound = outbound;
        this.objectMapper = objectMapper;
    }

    @Transactional
    public OrderResult create(CreateOrderCommand command) {
        Order order = orders.save(
                Order.create(command)
        );

        JsonNode payload = objectMapper.valueToTree(
                new CreateShipmentRequest(
                        order.getId(),
                        order.shippingAddress(),
                        order.items()
                )
        );

        outbound.enqueue(
                new EnqueueOutboundRequest(
                        UUID.randomUUID(),
                        "shipment:" + order.getId(),
                        "SHIPPING_PROVIDER",
                        "CREATE_SHIPMENT",
                        payload,
                        8
                )
        );

        return OrderResult.from(order);
    }
}

The writer uses the same PostgreSQL transaction and datasource as the order repository.

@Repository
public class OutboundRequestWriter {

    private final JdbcClient jdbc;

    public OutboundRequestWriter(JdbcClient jdbc) {
        this.jdbc = jdbc;
    }

    public void enqueue(EnqueueOutboundRequest command) {
        jdbc.sql("""
                INSERT INTO outbound_requests (
                    id,
                    business_key,
                    destination,
                    operation,
                    payload,
                    status,
                    attempt_count,
                    max_attempts,
                    next_attempt_at
                )
                VALUES (
                    :id,
                    :businessKey,
                    :destination,
                    :operation,
                    CAST(:payload AS jsonb),
                    'READY',
                    0,
                    :maxAttempts,
                    NOW()
                )
                ON CONFLICT (
                    destination,
                    operation,
                    business_key
                )
                DO NOTHING
                """)
                .param("id", command.id())
                .param("businessKey", command.businessKey())
                .param("destination", command.destination())
                .param("operation", command.operation())
                .param("payload", command.payload().toString())
                .param("maxAttempts", command.maxAttempts())
                .update();
    }
}

If the business row commits, the outbound request commits. If either insert fails, both roll back.

This guarantee exists only when both writes use the same local transaction manager and database. A normal Spring @Transactional method does not make two independently owned databases atomic.

Never Persist Arbitrary URLs or Authorization Headers

The database should store a destination key and operation name:

destination = SHIPPING_PROVIDER
operation   = CREATE_SHIPMENT

A registry maps that pair to approved application code.

public interface ExternalDestinationClient {

    String destination();

    String operation();

    DeliveryOutcome deliver(
            ClaimedOutboundRequest request
    );
}

This prevents a compromised row or client request from turning the dispatcher into an SSRF proxy.

Do not persist:

OAuth access tokens;
API keys;
cookies;
arbitrary forwarding headers;
user-supplied target URLs.

Authentication should be loaded at delivery time from a secret manager or OAuth client.

Claim Work Atomically

Several application replicas can poll the same table. A plain query allows every replica to read the same rows.

Claim work with a short transaction and FOR UPDATE SKIP LOCKED.

WITH candidates AS (
    SELECT id
    FROM outbound_requests
    WHERE (
        (
            status IN ('READY', 'RETRY_WAIT')
            AND next_attempt_at <= NOW()
        )
        OR
        (
            status = 'IN_PROGRESS'
            AND lease_until < NOW()
        )
    )
      AND attempt_count < max_attempts
    ORDER BY next_attempt_at, created_at
    FOR UPDATE SKIP LOCKED
    LIMIT :batch_size
)
UPDATE outbound_requests AS target
SET status = 'IN_PROGRESS',
    claim_token = :claim_token,
    claimed_by = :worker_id,
    lease_until = NOW()
        + make_interval(secs => :lease_seconds),
    attempt_count = target.attempt_count + 1,
    updated_at = NOW()
FROM candidates
WHERE target.id = candidates.id
RETURNING target.*;

SKIP LOCKED is appropriate for a queue-like table because workers may skip rows already claimed by another transaction. It is not a general-purpose consistent query mode.

The claim transaction should commit before any external HTTP call starts.

Why HTTP Must Be Outside the DB Transaction

Short transaction:
  claim rows
  commit

No DB transaction:
  perform HTTP calls

Short transaction:
  record success or failure
  commit

Holding a database transaction across a network call causes long row locks, connection-pool exhaustion, larger rollback scope, vacuum interference, and external latency becoming database latency.

The lease prevents a crashed worker from owning the row forever. Set the HTTP timeout below the lease or implement lease renewal for calls that can legitimately take longer.

Spring Boot Dependencies

<parent>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-parent</artifactId>
    <version>4.1.0</version>
    <relativePath />
</parent>

<properties>
    <java.version>25</java.version>
</properties>

<dependencies>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-restclient</artifactId>
    </dependency>

    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-data-jpa</artifactId>
    </dependency>

    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-actuator</artifactId>
    </dependency>

    <dependency>
        <groupId>org.postgresql</groupId>
        <artifactId>postgresql</artifactId>
        <scope>runtime</scope>
    </dependency>

    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-test</artifactId>
        <scope>test</scope>
    </dependency>
</dependencies>

For an imperative application, RestClient is clearer than using reactive WebClient and immediately calling .block().

Configure Timeouts Explicitly

Use Spring Boot's auto-configured RestClient.Builder so metrics and trace propagation can be applied.

@Configuration
@EnableConfigurationProperties(
        ShippingProviderProperties.class
)
public class ShippingClientConfiguration {

    @Bean
    RestClient shippingRestClient(
            RestClient.Builder builder,
            ShippingProviderProperties properties,
            ShippingAuthenticationInterceptor auth
    ) {
        HttpClient httpClient = HttpClient
                .newBuilder()
                .connectTimeout(
                        properties.connectTimeout()
                )
                .build();

        JdkClientHttpRequestFactory requestFactory =
                new JdkClientHttpRequestFactory(
                        httpClient
                );

        requestFactory.setReadTimeout(
                properties.readTimeout()
        );

        return builder
                .baseUrl(properties.baseUrl())
                .requestFactory(requestFactory)
                .requestInterceptor(auth)
                .build();
    }
}

Configure connection timeout, response timeout, total operation deadline, maximum response size, TLS verification, authentication refresh, and connection reuse.

A retry policy without a timeout policy can block a worker indefinitely.

Provider Client and Idempotency

@Component
public class ShippingProviderClient
        implements ExternalDestinationClient {

    private final RestClient client;

    public ShippingProviderClient(
            RestClient shippingRestClient
    ) {
        this.client = shippingRestClient;
    }

    @Override
    public String destination() {
        return "SHIPPING_PROVIDER";
    }

    @Override
    public String operation() {
        return "CREATE_SHIPMENT";
    }

    @Override
    public DeliveryOutcome deliver(
            ClaimedOutboundRequest request
    ) {
        try {
            ResponseEntity<CreateShipmentResponse> response =
                    client.post()
                            .uri("/v1/shipments")
                            .header(
                                    "Idempotency-Key",
                                    request.businessKey()
                            )
                            .contentType(
                                    MediaType.APPLICATION_JSON
                            )
                            .body(request.payload())
                            .retrieve()
                            .toEntity(
                                    CreateShipmentResponse.class
                            );

            CreateShipmentResponse body =
                    response.getBody();

            return DeliveryOutcome.succeeded(
                    body == null
                            ? null
                            : body.shipmentId()
            );
        } catch (RestClientResponseException exception) {
            return classifyResponse(exception);
        } catch (ResourceAccessException exception) {
            return DeliveryOutcome.transportFailure(
                    "NETWORK_OR_TIMEOUT",
                    exception.getMessage()
            );
        }
    }
}

The provider's idempotency contract must define:

key scope;
key retention period;
whether repeated requests return the original result;
behavior when the same key is reused with different payloads;
which status code represents a duplicate;
whether a request-status endpoint exists.

A generic X-Correlation-ID header is primarily for correlation. Do not assume a provider treats it as an idempotency key unless its contract says so.

Classify Results by Provider Contract

Outcome	Typical treatment
`2xx`	Success
Validation `400`	Permanent failure
Authentication `401/403`	Usually permanent until configuration changes; alert
Missing resource `404`	Operation-specific
Conflict `409`	Could be duplicate success or permanent conflict
Timeout `408`	Retry only when idempotency makes it safe
Rate limit `429`	Retry after provider delay
`5xx`	Usually retryable
Connection failure	Usually retryable
Read timeout after request sent	Potentially delivery unknown

Do not encode all providers in one universal status classifier. A 409 from one provider may mean “already created,” while another means “request can never succeed.”

Respect Retry-After when the provider sends it.

Retry with Exponential Backoff and Jitter

@Component
public class RetryDelayPolicy {

    private static final Duration BASE =
            Duration.ofSeconds(5);

    private static final Duration MAX =
            Duration.ofMinutes(30);

    public Instant nextAttemptAt(
            int attempt,
            Optional<Duration> providerDelay
    ) {
        if (providerDelay.isPresent()) {
            return Instant.now()
                    .plus(providerDelay.get());
        }

        long exponent = Math.min(
                Math.max(0, attempt - 1),
                12
        );

        long seconds = Math.min(
                MAX.toSeconds(),
                BASE.toSeconds() * (1L << exponent)
        );

        long jitter = ThreadLocalRandom
                .current()
                .nextLong(
                        Math.max(1, seconds / 4)
                );

        return Instant.now()
                .plusSeconds(seconds + jitter);
    }
}

Jitter prevents every failed request from retrying at the same instant after a provider recovers.

Maximum retries should not be the only stopping condition. Also consider maximum request age, business deadline, provider idempotency-key retention, user cancellation, and legal or financial cutoff times.

Separate State Transactions

The delivery processor does not own a long transaction.

@Component
public class OutboundDeliveryProcessor {

    private final DestinationRegistry destinations;
    private final OutboundStateService state;

    public OutboundDeliveryProcessor(
            DestinationRegistry destinations,
            OutboundStateService state
    ) {
        this.destinations = destinations;
        this.state = state;
    }

    public void process(
            ClaimedOutboundRequest request
    ) {
        ExternalDestinationClient client =
                destinations.clientFor(
                        request.destination(),
                        request.operation()
                );

        DeliveryOutcome outcome =
                client.deliver(request);

        switch (outcome) {
            case DeliveryOutcome.Succeeded success ->
                    state.markSucceeded(
                            request,
                            success.providerRequestId()
                    );

            case DeliveryOutcome.Retryable retryable ->
                    state.scheduleRetry(
                            request,
                            retryable
                    );

            case DeliveryOutcome.Permanent permanent ->
                    state.markPermanentFailure(
                            request,
                            permanent
                    );

            case DeliveryOutcome.Unknown unknown ->
                    state.markDeliveryUnknown(
                            request,
                            unknown
                    );
        }
    }
}

Each state method runs through a separate proxied bean.

@Service
public class OutboundStateService {

    private final JdbcClient jdbc;

    public OutboundStateService(JdbcClient jdbc) {
        this.jdbc = jdbc;
    }

    @Transactional
    public void markSucceeded(
            ClaimedOutboundRequest request,
            String providerRequestId
    ) {
        int updated = jdbc.sql("""
                UPDATE outbound_requests
                SET status = 'SUCCEEDED',
                    provider_request_id =
                        :provider_request_id,
                    succeeded_at = NOW(),
                    lease_until = NULL,
                    claim_token = NULL,
                    claimed_by = NULL,
                    updated_at = NOW()
                WHERE id = :id
                  AND status = 'IN_PROGRESS'
                  AND claim_token = :claim_token
                """)
                .param(
                        "provider_request_id",
                        providerRequestId
                )
                .param("id", request.id())
                .param(
                        "claim_token",
                        request.claimToken()
                )
                .update();

        if (updated != 1) {
            throw new StaleClaimException(
                    request.id()
            );
        }
    }
}

The claim_token guard prevents an old worker from updating a row after its lease expired and a new worker reclaimed it.

It cannot undo an external side effect already performed by the old worker. That is why the provider idempotency key remains necessary.

Dispatcher with Bounded Parallelism

Virtual threads are appropriate for many concurrent blocking HTTP calls. They do not make the provider, database, or connection pool unlimited.

Create one managed executor:

@Configuration
public class OutboundExecutionConfiguration {

    @Bean(
        name = "outboundExecutor",
        destroyMethod = "close"
    )
    ExecutorService outboundExecutor() {
        return Executors
                .newVirtualThreadPerTaskExecutor();
    }
}

Claim a bounded batch and wait for its tasks.

@Component
public class OutboundDispatcher {

    private final OutboundClaimService claims;
    private final OutboundDeliveryProcessor processor;
    private final ExecutorService executor;

    private final String workerId =
            UUID.randomUUID().toString();

    public OutboundDispatcher(
            OutboundClaimService claims,
            OutboundDeliveryProcessor processor,
            @Qualifier("outboundExecutor")
            ExecutorService executor
    ) {
        this.claims = claims;
        this.processor = processor;
        this.executor = executor;
    }

    @Scheduled(
        fixedDelayString =
            "${outbound.dispatch.fixed-delay:1s}"
    )
    public void dispatch() {
        UUID claimToken = UUID.randomUUID();

        List<ClaimedOutboundRequest> batch =
                claims.claim(
                        workerId,
                        claimToken,
                        50,
                        Duration.ofMinutes(2)
                );

        List<Future<?>> futures = batch.stream()
                .map(request ->
                        executor.submit(
                                () -> processor.process(
                                        request
                                )
                        )
                )
                .toList();

        for (Future<?> future : futures) {
            try {
                future.get();
            } catch (InterruptedException exception) {
                Thread.currentThread().interrupt();
                return;
            } catch (ExecutionException exception) {
                // The row remains leased and can be
                // recovered after lease expiry.
            }
        }
    }
}

The maximum active calls in this example is bounded by the claimed batch size per dispatcher replica. For stricter global limits, combine batch size with provider rate limits and a distributed permit mechanism.

Spring Boot can enable virtual threads globally:

spring:
  threads:
    virtual:
      enabled: true

  main:
    keep-alive: true

Virtual threads help blocked I/O workloads. They do not make CPU-heavy transformations faster, and they do not remove downstream capacity limits.

Why Detached Scheduler Tasks Are Unsafe

This pattern is dangerous:

@Transactional
@Scheduled(...)
public void dispatch() {
    List<OutboundRequest> rows =
            repository.findPending();

    rows.forEach(row ->
            CompletableFuture.runAsync(
                    () -> process(row)
            )
    );
}

Problems:

the scheduler transaction ends while detached tasks continue;
JPA entities may be detached;
multiple replicas can read the same rows;
the scheduler returns before work finishes;
task exceptions may be logged but not persisted;
@Transactional methods called inside the same class may bypass the proxy;
there is no stale-claim recovery;
the query may load an unbounded backlog.

A durable claim must be committed before execution begins.

Crash Windows

Crash after claim, before HTTP

The row remains IN_PROGRESS. After lease_until, another worker can reclaim it.

Crash after provider success, before DB success update

The row is retried after lease expiry.

This is the most important duplicate window. A stable provider idempotency key should return the original result rather than create another external effect.

Timeout after provider success

The client cannot know whether the operation completed. Use:

provider status lookup by idempotency key;
retry with the same idempotency key;
reconciliation;
DELIVERY_UNKNOWN and operator review.

Do not mark the request SUCCEEDED merely because it was sent, and do not blindly retry a non-idempotent operation.

Crash after retry state update

The row remains durable with its next-attempt time and will be picked up later.

Optional Kafka Coordination

A database poller is sufficient for many systems. Kafka becomes useful when:

dispatch demand has large spikes;
several processors need status events;
dispatch is owned by a separate service;
the organization already operates Kafka reliably;
a wake-up signal should reduce database polling latency.

Kafka does not remove the need for the durable request row.

Do Not Publish Kafka Only with `AFTER_COMMIT`

This sequence has a loss window:

PostgreSQL commit succeeds
AFTER_COMMIT listener starts
JVM crashes before Kafka acknowledges send
No Kafka record exists

@TransactionalEventListener defaults to AFTER_COMMIT, but the listener invocation itself is not durable. KafkaTemplate.send() also returns an asynchronous future.

Use one of these approaches:

Transactional outbox

Write an outbox row in the same PostgreSQL transaction, then publish it with an outbox relay.

CDC

Use Debezium to capture committed outbound-request rows or outbox rows and publish them to Kafka.

The Kafka message should contain a stable request ID:

{
  "requestId": "8c777b7c-f3d9-4f52-a694-eab9b3a31cf0"
}

The consumer loads and claims the database row. The Kafka message is a wake-up signal, not proof that the external request still needs execution.

A duplicate Kafka message is harmless when the database state machine and claim token are authoritative.

Delivery Events

After changing the database state, the service may publish status events:

OutboundDeliverySucceeded
OutboundDeliveryRetryScheduled
OutboundDeliveryFailed
OutboundDeliveryUnknown

Use a second outbox for these events if they must not be lost.

Do not call Kafka directly and assume a local database status update plus Kafka publish are atomic.

Security

An outbound gateway handles sensitive integration data.

Destination allowlist

Only preconfigured clients may be selected. Never execute a URL stored directly from user input.

Secrets

Load credentials at dispatch time. Do not persist bearer tokens or API keys with the request.

Payload retention

Define:

which fields are stored;
whether values are encrypted;
who can query failed payloads;
retention after success;
redaction in logs;
deletion obligations.

Response handling

Do not store full external error bodies without limits. They can contain personal data, HTML, or very large payloads.

TLS

Verify hostnames and certificates. Do not disable TLS verification to make a development endpoint work.

Observability

Track:

count by status and destination;
oldest READY or RETRY_WAIT age;
claim rate;
delivery success and failure rate;
HTTP latency by destination and operation;
retries by reason;
rate-limit responses;
DELIVERY_UNKNOWN count;
expired lease count;
dead-request count;
provider reconciliation backlog;
executor in-flight count;
database connection-pool usage.

Use bounded metric labels:

destination
operation
outcome
error_category
http_status_class

Do not use request IDs or business keys as metric labels.

Useful log fields:

outbound.request_id
outbound.destination
outbound.operation
outbound.attempt
outbound.status
provider.request_id
traceId

Never log the complete request payload by default.

Administrative Operations

Operators need safe actions:

retry a DEAD request;
reconcile a DELIVERY_UNKNOWN request;
cancel a pending request;
inspect attempt history;
filter by business key;
view the last sanitized error;
pause one destination;
reduce concurrency during provider degradation.

Record every manual action in an audit table.

Do not let an operator retry a non-idempotent unknown delivery without confirmation and reconciliation.

Attempt History

Keeping only the latest error can make incidents difficult to explain.

CREATE TABLE outbound_attempts (
    id UUID PRIMARY KEY,
    request_id UUID NOT NULL,
    attempt_number INTEGER NOT NULL,
    claim_token UUID NOT NULL,
    started_at TIMESTAMPTZ NOT NULL,
    completed_at TIMESTAMPTZ,
    outcome VARCHAR(40),
    http_status INTEGER,
    error_code VARCHAR(100),
    error_message VARCHAR(1000),
    provider_request_id VARCHAR(250),
    UNIQUE (request_id, attempt_number)
);

Attempt history helps answer:

Was the provider called?
Did all failures have the same cause?
Was Retry-After respected?
Which worker owned the attempt?
Did the provider return an identifier before a timeout?

Sanitize stored errors.

Testing the Failure Windows

Use PostgreSQL Testcontainers and an HTTP stub such as WireMock or MockWebServer.

Transaction atomicity

fail outbound insertion;
verify the business row also rolls back;
fail business persistence;
verify no outbound request exists.

Concurrent claiming

run several dispatcher instances;
verify each row has one active claim token;
verify locked rows are skipped.

Stale lease

claim a row and stop the worker;
wait for lease expiry;
verify another worker reclaims it.

Provider success followed by local crash

let the provider record success;
terminate before markSucceeded;
retry with the same idempotency key;
verify one provider-side resource exists.

Ambiguous timeout

make the provider complete the request;
delay the response beyond the client timeout;
verify the request becomes retryable only when the provider idempotency contract is enabled;
otherwise verify DELIVERY_UNKNOWN.

Permanent failure

return a validation error;
verify no automatic retry occurs.

Rate limiting

return 429 and Retry-After;
verify the next attempt respects the provider delay.

Retry exhaustion

return repeated retryable failures;
verify the request enters DEAD;
verify an alert or status event is emitted.

Graceful shutdown

stop the service during active delivery;
verify completed results are persisted;
verify unfinished claims are recoverable after lease expiry.

Troubleshooting

Rows remain `READY`

Check:

scheduler is enabled;
dispatch query index exists;
next_attempt_at is not in the future;
no destination is paused;
claim transaction commits;
database time zone is correct.

Rows remain `IN_PROGRESS`

Check:

lease_until;
dispatcher crashes;
state-update failures;
transaction proxy configuration;
HTTP timeout versus lease duration;
stale-claim recovery query.

Duplicates appear at the provider

Check:

provider actually supports idempotency;
the same stable key is reused for retries;
key retention has not expired;
payload does not change for the same key;
the success-before-local-update crash window;
manual retries.

Dispatcher throughput is low

Check:

provider latency and rate limits;
batch size;
request timeout;
virtual-thread pinning;
database claim latency;
connection-pool saturation;
destination concurrency policy.

Increasing virtual-thread count does not increase provider quota.

Kafka wake-up events are missing

Check whether events are produced through a durable outbox or CDC. An AFTER_COMMIT listener alone cannot guarantee publication after a process crash.

Decision Checklist

Before production:

Is the business intent stored atomically?
Is the business key unique?
Are destinations allowlisted?
Are secrets excluded from rows?
Are claims protected with SKIP LOCKED?
Is HTTP outside the DB transaction?
Is stale-claim recovery implemented?
Is request timeout shorter than the lease?
Does the provider support idempotency?
What does a timeout mean for this operation?
Which responses are retryable?
Is Retry-After respected?
Is retry concurrency bounded?
Are unknown deliveries reconciled?
Is Kafka publication durable?
Can operators inspect and replay safely?
Have crash windows been tested?

Conclusion

A reliable outbound gateway is a persisted state machine around an unreliable network boundary.

The essential design is:

save business data and outbound intent in one local transaction;
deduplicate scheduling with a database constraint;
claim bounded work through FOR UPDATE SKIP LOCKED;
commit the claim before making HTTP calls;
use a configured imperative client with strict timeouts;
reuse one provider idempotency key across retries;
classify permanent, retryable, and ambiguous outcomes separately;
persist retry timing with exponential backoff and jitter;
guard updates with a claim token;
recover stale leases;
use Kafka only through an outbox or CDC boundary;
monitor backlog age, unknown outcomes, and exhausted retries;
test crashes before and after every durable boundary.

The pattern does not make an external API transactional. It makes your service honest about uncertainty and durable enough to recover from it.

What This Pattern Can Guarantee

The Core State Machine

Database Schema

Map JSON with Hibernate

Store Intent in the Business Transaction

Never Persist Arbitrary URLs or Authorization Headers

Claim Work Atomically

Why HTTP Must Be Outside the DB Transaction

Spring Boot Dependencies

Configure Timeouts Explicitly

Provider Client and Idempotency

Classify Results by Provider Contract

Retry with Exponential Backoff and Jitter

Separate State Transactions

Dispatcher with Bounded Parallelism

Why Detached Scheduler Tasks Are Unsafe

Crash Windows

Crash after claim, before HTTP

Crash after provider success, before DB success update

Timeout after provider success

Crash after retry state update

Optional Kafka Coordination

Do Not Publish Kafka Only with AFTER_COMMIT

Transactional outbox

CDC

Delivery Events

Security

Destination allowlist

Secrets

Payload retention

Response handling

TLS

Observability

Administrative Operations

Attempt History

Testing the Failure Windows

Transaction atomicity

Concurrent claiming

Stale lease

Provider success followed by local crash

Ambiguous timeout

Permanent failure

Rate limiting

Retry exhaustion

Graceful shutdown

Troubleshooting

Rows remain READY

Rows remain IN_PROGRESS

Duplicates appear at the provider

Dispatcher throughput is low

Kafka wake-up events are missing

Decision Checklist

Conclusion

Official References

Do Not Publish Kafka Only with `AFTER_COMMIT`

Rows remain `READY`

Rows remain `IN_PROGRESS`