The same message is processed twice — at-least-once delivery and idempotency

What this error actually means

There is no exception for this one, which is precisely why it survives in production for months. The system works; it just occasionally does the same thing twice, and the evidence is a duplicate row, a second email, or a customer charged twice — discovered by a human long after the log lines have rotated away.

Redelivery is not a fault. Every broker in mainstream use — Kafka, SQS, RabbitMQ, Pub/Sub — delivers at least once, because the alternative is losing messages. The acknowledgement is a second network round trip after the work, and any failure in that window (a crash, a rebalance, a lost ack, a visibility timeout expiring while you were still working) leaves the broker with no evidence that the work happened. Its only safe move is to deliver again.

So the interesting question is never "why was it delivered twice" but "what in my handler is not safe to run twice". Handlers divide cleanly. A `SET status = "shipped"` is naturally idempotent and can run a thousand times. A `balance = balance - 10`, an `INSERT` without a unique constraint, a `POST /charges`, or an email send are not, and each needs an explicit key that lets the second execution recognise the first one.

The subtlety that catches careful teams: two deliveries can be *concurrent*, not sequential. During a rebalance or a visibility-timeout expiry the original worker is often still running. Any deduplication built as "select, then decide, then write" has a race between the select and the write, and under exactly the conditions that cause redelivery, both workers will pass the check.

How to diagnose it

SELECT order_id, count(*), max(created_at) - min(created_at) AS gap
FROM shipments
GROUP BY order_id HAVING count(*) > 1
ORDER BY 2 DESC LIMIT 20;

aws sqs receive-message --queue-url "$Q" --attribute-names ApproximateReceiveCount --max-number-of-messages 1

The fix

Give every message a stable business identity chosen by the producer — an order id, an event id, a payment intent id — and make it part of the payload. Broker-assigned identifiers are not enough, because a double publish produces two of them for the same event.

Enforce uniqueness in the database rather than in application logic. A `UNIQUE` constraint on the business key, combined with `INSERT ... ON CONFLICT DO NOTHING`, is atomic and correct under concurrency; a preliminary `SELECT` is neither. Treat the conflict as success and acknowledge the message.

Where the effect is an update rather than an insert, make the write conditional on state so the second execution is a no-op: `UPDATE orders SET status = "shipped" WHERE id = $1 AND status = "packed"`. Check the affected-row count to know whether you were first.

For external calls, pass an idempotency key derived from the business identity and let the provider deduplicate. Record your own outcome for that key in the same transaction as the rest of the work, so a replay finds the recorded result instead of calling again.

Keep the deduplication record and the business effect in one transaction. Writing the effect and then marking the message as seen leaves a window where a crash between them produces a duplicate on replay — which is the same class of bug as the dual write this whole family of failures comes from.

// Not deduplication: both workers pass the check before either inserts.
if (shipmentRepo.findByOrderId(evt.orderId()).isEmpty()) {
    shipmentRepo.save(new Shipment(evt.orderId()));
    carrier.dispatch(evt.orderId());
}

// Atomic claim first; only the winner performs the effect.
@Transactional
public void handle(OrderPacked evt) {
    int claimed = jdbc.update("""
            INSERT INTO processed_events (event_id, consumer, processed_at)
            VALUES (?, "shipment-consumer", now())
            ON CONFLICT (event_id, consumer) DO NOTHING
            """, evt.eventId());
    if (claimed == 0) return;              // a previous delivery already did this

    shipmentRepo.save(new Shipment(evt.orderId()));
    carrier.dispatch(evt.orderId(), evt.eventId());   // key passed downstream too
}

-- the constraint that makes the claim atomic rather than advisory
CREATE TABLE processed_events (
    event_id    text NOT NULL,
    consumer    text NOT NULL,
    processed_at timestamptz NOT NULL,
    PRIMARY KEY (event_id, consumer)
);

How to stop it coming back

• Write down the delivery semantics of every consumer in the service README: at-least-once with an idempotency key, or at-least-once with a naturally idempotent effect. There is no third option that is safe
• Add a contract test that delivers each message type twice, concurrently, and asserts the end state — this is cheap and catches regressions when a handler grows a new side effect
• Prune the deduplication table on a schedule with a retention window longer than your maximum redelivery delay, so it does not become the next incident
• Make producers idempotent where the broker supports it (`enable.idempotence=true` for Kafka) so double publishes stop at the source
• Alarm on duplicate business keys as a data-quality metric; the failure is silent otherwise and the log evidence expires before the ticket arrives

FAQ

Related

Other errors engineers hit next to this one

• celery.exceptions.WorkerLostError: Worker exited prematurely
• requests.exceptions.ReadTimeout: HTTPSConnectionPool read timed out
• UnicodeDecodeError: 'utf-8' codec can't decode byte
• AssertionError: daemonic processes are not allowed to have children
• [CRITICAL] WORKER TIMEOUT (pid:1234)
• Mutable default argument retains state across calls
• FATAL ERROR: Reached heap limit Allocation failed
• Unhandled promise rejection crashes the process