A webhook was delivered twice and the customer was charged twice

What this error actually means

Webhook delivery is at-least-once, and the retry policy is the provider’s, not yours. Any delivery where the provider does not see a timely 2xx is retried — including the case where your handler completed the work perfectly and the response was lost, slow, or dropped by a proxy. The provider cannot distinguish "not processed" from "processed, acknowledgement lost", so it retries.

This means a slow handler is an active cause of duplicates rather than merely a performance concern. Most providers time out delivery in the low tens of seconds. A handler that does the payment reconciliation, the invoice write, the email, and the analytics call inline will sit near that limit under load, and every delivery that crosses it is retried while the first execution is still running — producing two concurrent executions of the same event, not two sequential ones.

The financial damage comes from a second property: most webhook side effects are not naturally idempotent. Appending a ledger entry, incrementing a balance, extending a subscription period, or issuing a refund are all "do it again and it happens again" operations. The provider gave you a stable event identifier precisely so you can refuse the second execution, and ignoring it is what converts an ordinary retry into a double charge.

Ordering is the companion problem. Retries interleave, so `payment.captured` can arrive after `charge.refunded`, and a subscription’s `updated` events can arrive out of sequence. Deduplicating by event id fixes repeats but not inversions; the durable answer is to key the effect on the business object and apply it only when the incoming state is newer than the stored state.

How to diagnose it

stripe events resolve evt_3PqLk9aBcD

awk '$7 ~ /webhooks/ {print $NF}' access.log | sort -n | awk '{a[NR]=$1} END {print "p50", a[int(NR*0.5)], "p99", a[int(NR*0.99)], "max", a[NR]}'

SELECT payment_id, count(*), sum(amount_minor)
FROM ledger_entries
WHERE created_at > now() - interval '7 days'
GROUP BY payment_id HAVING count(*) > 1;

The fix

Acknowledge fast and process asynchronously. Validate the signature, persist the raw event with its provider event id under a unique constraint, return 200, and let a worker do the rest. This collapses handler latency to a single insert and removes the timeout-driven retries that cause most duplicates in the first place.

Make the provider event id the primary key of the stored event, and treat a conflict as success. `INSERT ... ON CONFLICT DO NOTHING` followed by a check of the affected-row count is atomic, so of two concurrent retries exactly one proceeds. Never implement this as a `SELECT` followed by an `INSERT`.

Apply the effect in the same transaction as the claim, or make the worker itself idempotent on the business key. If the claim commits and the process dies before the effect, the event is marked handled and the work is lost — a silent under-count that is harder to find than a duplicate.

Guard the business object against out-of-order events with a version or state condition, not just against repeats: extend a subscription period only if the new period end is later than the stored one, and apply a capture only if the payment is not already captured or refunded.

Verify the signature and reject unsigned deliveries before any of this. Deduplication logic keyed on an attacker-supplied event id is a replay vector, and the signature check is what makes the id trustworthy.

// Slow, unconditional, and duplicated on every provider retry.
@PostMapping("/webhooks/payments")
public ResponseEntity<Void> handle(@RequestBody String body) {
    var evt = verifyAndParse(body);
    ledger.credit(evt.paymentId(), evt.amountMinor());   // appends a row
    invoices.markPaid(evt.paymentId());
    mailer.sendReceipt(evt.customerId());                // ~2s on a good day
    return ResponseEntity.ok().build();
}

// Claim atomically, acknowledge immediately, do the work once, out of band.
@PostMapping("/webhooks/payments")
public ResponseEntity<Void> handle(@RequestBody String body,
                                   @RequestHeader("Stripe-Signature") String sig) {
    var evt = verifyAndParse(body, sig);                 // throws -> 400, no retry loop
    int claimed = jdbc.update("""
            INSERT INTO webhook_events (event_id, type, payload, received_at)
            VALUES (?, ?, ?::jsonb, now())
            ON CONFLICT (event_id) DO NOTHING
            """, evt.id(), evt.type(), body);
    if (claimed == 1) {
        worker.enqueue(evt.id());                        // same transaction as the claim
    }
    return ResponseEntity.ok().build();                  // sub-50ms, always
}

-- and the effect itself refuses to run twice or backwards
UPDATE payments
   SET status = "captured", captured_at = now()
 WHERE id = $1
   AND status = "authorized";        -- 0 rows => already captured or refunded

How to stop it coming back

• Alarm on webhook handler p99 latency against the provider’s delivery timeout; this is the leading indicator of duplicates
• Make the unique constraint on the provider event id a schema-level requirement for every webhook table, reviewed like any other invariant
• Reconcile daily against the provider’s settlement or event export and alert on divergence — duplicates and losses are both silent without it
• Keep webhook secrets per environment so a stale staging deployment cannot process production events
• Replay a recorded delivery twice in CI, including concurrently, and assert one effect

FAQ

Related

Other errors engineers hit next to this one

• Undici / fetch connections remain occupied
• ERR_UNHANDLED_REJECTION in a worker thread
• command not found in a script that works interactively
• Permission denied when executing a script
• bad interpreter: No such file or directory with CRLF
• Argument list too long
• Too many open files
• Out of memory: Killed process