Transactional Outbox Architecture: Separating Business Commits from Notification Delivery

An API can successfully save a business change and still lose the notification that should follow it. Sending the notification first reverses the risk: the external effect may happen even if the business transaction later rolls back.

I kept the business write and outbox insert in one PostgreSQL transaction, then separated Kafka publication from notification delivery. This made the delivery obligation durable without making the API wait for the external provider.

A transaction port kept framework dependencies out of the domain. Explicit message states tracked progress across the publisher and consumer.

The boundary that asynchronous execution did not solve

The requirement was not simply to return from the API sooner. It was to preserve the obligation to perform later work after the API's business transaction committed.

Three superficially similar approaches have different failure boundaries:

An after-commit listener avoids sending for a transaction that rolls back, but does not make its in-memory work durable. That is the deciding difference when loss after commit is unacceptable. Spring asynchronous events explains the listener boundary in more detail.

Nor does placing a Kafka send inside a Spring database transaction make both systems one atomic resource. Spring Kafka documents synchronized transaction arrangements and the possibility of a secondary commit failing after the primary one has committed. See its transaction reference.

The outbox chooses a narrower guarantee that the application can actually enforce: the business write and the saved publication intent commit together in one database.

Three roles, three meanings of success

I separated these responsibilities:

API process
  validate command
  begin PostgreSQL transaction
    persist business change
    insert immutable event intent
  commit
  return business result

publisher process
  discover committed event intent
  claim work
  send to Kafka
  record publication outcome

notification consumer process
  receive Kafka record
  claim delivery
  call external provider
  record delivery outcome
  advance Kafka progress under the configured policy

The API owns the business transaction. The publisher owns transport to Kafka. The consumer owns the attempt to perform the external effect. Calling all three outcomes “sent” hides useful distinctions.

For example, an API success can truthfully mean that the business change and notification obligation are committed. It cannot mean that the recipient has received a notification. Kafka acknowledgment means the broker accepted the configured publication; provider acceptance is another result, and end-user receipt may be yet another.

This distinction also changes the user-facing flow. If a notification may arrive later, the application needs a pending state, a safe retry or resend policy, and a way to diagnose delays. Those product decisions are consequences of asynchronous delivery, not evidence of a broker malfunction.

Write the event in the same transaction

An outbox event needs a stable identity, a routing key, an explicit type, and enough immutable data for its consumer. A conceptual envelope might be:

{
  "eventId": "00000000-0000-4000-8000-000000000001",
  "aggregateKey": "sample-request-42",
  "eventType": "NotificationRequested",
  "schemaVersion": 1,
  "payload": {
    "template": "request-received",
    "deliveryReference": "sample-recipient-reference"
  }
}

The event ID identifies this logical event across relay retries. The aggregate key groups related events for routing. The schema version describes the contract, not the application build number.

The payload should be a deliberate integration contract. Serializing the current database entity can expose internal fields and couple consumers to persistence changes. Reloading mutable business data later may also turn the event into a different historical fact. If the payload uses a reference, explicitly define whether the consumer is meant to read current or historical data and how it handles missing references.

A minimal transaction example makes the atomicity visible:

BEGIN;

INSERT INTO business_request (request_id, state)
VALUES (:request_id, 'accepted');

INSERT INTO event_intent (
    event_id, aggregate_key, event_type, schema_version, payload
)
VALUES (
    :event_id, :aggregate_key, 'NotificationRequested', 1, :payload
);

COMMIT;

The table names and named bindings are illustrative. Both writes must use the same local PostgreSQL transaction. A separate connection, a new transaction for the event insert, or a swallowed insertion failure can break the guarantee.

Keep transaction mechanics behind a port

The project used a hexagonal, multi-module structure. Domain code defined ports, application code composed the business flow, and infrastructure adapters supplied persistence and transaction behavior. Instead of putting Spring's @Transactional annotation on the use case, I exposed this Kotlin contract:

interface TxExecutorPort {
    fun <T> required(block: () -> T): T
    fun <T> requiredNew(block: () -> T): T
}

A simplified use case shows how the port keeps the two writes together:

fun accept(command: RequestCommand): UUID = tx.required {
    val requestId = UUID.randomUUID()
    requests.insert(requestId, command)
    eventIntents.insert(notificationFor(requestId, command))
    requestId
}

required must join or create the transaction that contains both writes. Using requiredNew for the outbox insert would separate their commit outcomes and defeat the intended boundary.

The adapter owns the actual transaction manager and propagation settings. Spring's programmatic transaction support provides callback-based mechanisms such as TransactionTemplate for implementing that responsibility. The interface alone does not create a transaction: persistence adapters must participate in the same underlying resource.

This adds a port and adapter to maintain, but keeps the domain independent of Spring APIs and makes transaction scope visible in the use case. It was an architectural choice in the project, rather than a requirement of the outbox pattern itself.

The code contains no Kafka or provider call. An unavailable broker therefore does not have to fail this local transaction immediately. That is isolation of the request path, not unlimited outage tolerance: backlog storage and delivery deadlines still impose limits.

Why separate the publisher from the notification consumer?

I separated database polling from external notification I/O. The common producer selected its persistence adapter through worker.mode, while the notification consumer handled the delivery side. The same producer code supported notification and domain modes.

The server handoff document explicitly distinguished their status: notification processing was the active path, the domain producer was executable but unused, and the domain consumer was still planned at that point. That limitation applies to the domain extension, not to the implemented notification pipeline.

The separation is useful when the relay should keep moving committed intent into Kafka while provider calls have their own concurrency limits and error handling. It also prevents the API process from owning long-running notification attempts.

The price is more state and more places to observe progress. Database pending age, Kafka lag, and provider failures measure different delays. An empty outbox backlog can coexist with a stalled consumer, and low Kafka lag can coexist with quarantined notifications.

Do not infer that worker separation requires a new repository, image, or service for each role. A shared artifact with explicit runtime mode can be sufficient. Validate that a process starts only its intended polling loop or listener, fails on unsupported configuration, and receives only the permissions that role needs. These are deployment checks, not reasons to publish internal hostnames or account configuration.

The outbox is not the only possible structure

Kafka was part of this implementation, but a small notification system may not need a broker between the durable table and a delivery worker. If one worker can directly process the table and the application does not need Kafka retention, fan-out, or consumer isolation, the extra hop may cost more than it provides.

A polling relay is also not the only way to move committed rows to Kafka. Debezium's outbox event router turns captured outbox inserts into event records. That shifts capture and progress tracking to CDC infrastructure. It introduces connector operations and a different table contract; it does not remove the possibility of replay or the consumer's external side-effect problem.

I used application workers for this pipeline. The alternatives change who owns capture, scheduling, and recovery.

Verify the structure before tuning the workers

To validate this architecture in a runnable application, check the transaction and worker boundaries separately:

1. Force the outbox insert to fail after the business write has executed. Verify that neither row commits.
2. Commit a valid request with the publisher stopped. Verify that the event remains in PostgreSQL and can be discovered after the publisher starts.
3. Make Kafka unavailable. Check that committed intent remains pending and that backlog limits and alerts behave as designed.
4. Stop the consumer while publication continues. Verify that publisher success is not reported as completed notification delivery.
5. Start each worker mode separately. Confirm that only the intended role runs and that invalid modes fail visibly.
6. Replay the same event ID. Verify that consumer completion policy handles duplicates rather than treating every delivery as new work.

Use real database transaction behavior for the first two checks. A mock verifying two repository calls cannot show that both writes committed or rolled back together. The later checks need broker and worker integration as well. Record database outcomes and worker behavior separately so an API success cannot conceal a failed handoff.

Accept the operational cost deliberately

This structure fits a durable asynchronous obligation: business state must commit even when downstream delivery is temporarily unavailable, and later repair matters more than immediate provider completion. It is less suitable when the business operation itself cannot succeed without a synchronous external result.

It also requires an owner for stalled intent, retry exhaustion, payload evolution, retention, and replay. A failed event must remain actionable, and deleting old data must not erase the evidence needed to interpret a retry.

The architecture solves the first gap: a committed business change cannot lose its saved event merely because the API process exits. The remaining guarantees require separate designs. Publisher recovery covers concurrent claims and crashes around Kafka sends. Consumer recovery covers offsets, claim states, and ambiguous external outcomes. Each begins at its own boundary because a durable handoff is only as useful as the recovery policy on the receiving side.