Dual write: the database committed but the event was never published

What this error actually means

Two independent systems, two independent commits, no shared transaction. Whatever order you choose, there is a window in which one has accepted the write and the other has not, and a crash or a timeout inside that window leaves the two permanently disagreeing. No amount of careful ordering removes the window; ordering only decides which kind of inconsistency you get.

Commit the database first and publish second, and a publish failure gives you an order that exists but was never announced — nothing ships, no email is sent, and the downstream read model simply lacks the record. Publish first and commit second, and a rollback gives you an event for an order that does not exist; consumers then act on a phantom, and the harder failure mode is that they may create real, non-reversible effects from it.

The reason the timeout in the log above is so dangerous is that a publish timeout is ambiguous. The broker may well have persisted the record and only the acknowledgement was lost. So a retry can double-publish and an abandonment can lose the event — from the producer’s position those cases are indistinguishable, which is why "just retry the publish" is not a fix.

The pattern that removes the window is to stop writing to two systems. Write the business row and a row describing the event into the same database transaction, so they commit or roll back together, then let a separate process move outbox rows to the broker. Publication becomes a retriable, at-least-once activity with a durable record of intent — and the only remaining requirement is that consumers tolerate duplicates, which they must anyway.

How to diagnose it

SELECT o.id, o.created_at
FROM orders o
LEFT JOIN order_projection p ON p.order_id = o.id
WHERE o.created_at < now() - interval '15 minutes'
  AND p.order_id IS NULL
ORDER BY o.created_at;

grep -E 'TimeoutException|NotLeaderOrFollowerException|RecordTooLargeException|failed to publish' app.log | wc -l

grep -E 'acks|enable.idempotence|delivery.timeout.ms' src/main/resources/application*.yml

The fix

Adopt the transactional outbox. In the same transaction as the business change, insert a row into an `outbox` table containing the aggregate id, event type, payload, and a creation timestamp. The transaction is the atomicity boundary, so there is no longer any window where one side committed and the other did not.

Publish from the outbox with a separate relay: either a poller that claims rows with `FOR UPDATE SKIP LOCKED`, publishes, and marks them sent, or change data capture reading the write-ahead log (Debezium is the usual choice) so no polling is needed at all. Either way publication is retriable because the intent is durable.

Accept that the relay is at-least-once and make consumers idempotent. A crash after publishing and before marking the row sent republishes the event. That is the correct trade: duplicates you can deduplicate, versus losses you cannot detect.

Preserve per-aggregate order by keying the published record on the aggregate id and having the relay process rows in id order per aggregate. An outbox drained by a multi-threaded relay with no key affinity reintroduces the out-of-order problem you may have just fixed elsewhere.

Set the producer to `acks=all` with `enable.idempotence=true` and a finite `delivery.timeout.ms` so the relay learns about failures reliably and its own retries do not reorder or duplicate within the broker. Then alert on outbox lag — the age of the oldest unsent row — which is the one metric that makes this whole mechanism observable.

// Dual write: two commits, no atomicity, and a catch that loses the event.
@Transactional
public void confirm(String orderId) {
    orders.updateStatus(orderId, CONFIRMED);
    try {
        kafka.send("order-events", orderId, payload(orderId)).get();
    } catch (Exception e) {
        log.error("failed to publish {}", orderId, e);   // silently divergent
    }
}

// Outbox: one transaction, one atomicity boundary.
@Transactional
public void confirm(String orderId) {
    orders.updateStatus(orderId, CONFIRMED);
    jdbc.update("""
            INSERT INTO outbox (id, aggregate_id, type, payload, created_at)
            VALUES (?, ?, "order.confirmed", ?::jsonb, now())
            """, UUID.randomUUID(), orderId, payload(orderId));
}

-- the relay: concurrent-safe claim, ordered per aggregate, retriable
WITH claimed AS (
  SELECT id, aggregate_id, type, payload
  FROM outbox
  WHERE sent_at IS NULL
  ORDER BY created_at
  FOR UPDATE SKIP LOCKED
  LIMIT 100
)
SELECT * FROM claimed;
-- publish, then:
UPDATE outbox SET sent_at = now() WHERE id = ANY($1);

How to stop it coming back

• Make "no publish inside a service method that also writes the database" a review rule; the outbox insert is the only allowed form
• Alert on the age of the oldest unsent outbox row rather than on publish exceptions — it catches a stopped relay, a stuck row, and a poison payload alike
• Keep a permanent reconciliation job comparing source counts with derived counts; the outbox makes divergence rare, not impossible
• Delete or archive sent outbox rows on a schedule, and index `(sent_at, created_at)` so the claim query stays cheap as the table ages
• Test the relay’s crash window explicitly: kill it between publish and mark-sent and assert the consumer end state is unchanged

FAQ

Related

Other errors engineers hit next to this one

• CROSSSLOT Keys in request don’t hash to the same slot
• MOVED and ASK replies from Redis Cluster
• Redis clients stall during KEYS on a large keyspace
• A Redis lock lease expires while the worker still runs
• HikariPool-1 - Connection is not available, request timed out
• java.lang.OutOfMemoryError: Java heap space
• java.lang.OutOfMemoryError: Metaspace
• java.lang.OutOfMemoryError: GC overhead limit exceeded