Distributed systems

Message ordering guarantees: interview questions and how to answer them

Q: What ordering does Kafka actually guarantee?

Total order within a partition, and nothing across partitions. Messages with the same key hash to the same partition, so per-key order holds provided the producer is idempotent and one consumer instance processes that partition sequentially. Consumers read partitions independently, so interleaving across partitions is arbitrary.

Q: You need per-customer ordering and high throughput. How do you get both?

Key by customer id so each customer’s events share a partition, and scale by partition count rather than by threads per partition. If one customer is large enough to dominate a partition, either shard that customer’s key with a sub-key when its events are independent, or switch to version-conditional writes so ordering stops being required at all.

Q: An event arrives whose predecessor has not. What should the consumer do?

It depends on whether the handler is convergent. With versioned conditional writes, apply it — the late predecessor will later fail its own `version <` check and be discarded harmlessly. With a strict state machine, park it in a keyed buffer with a timeout and apply it once the gap closes; on timeout, escalate rather than silently dropping. What it must not do is throw, because that routes a valid event to the dead-letter queue.

Q: Why not just use one partition to get global ordering?

Because a single partition is a single-threaded pipeline: one broker leader for all writes, one consumer for all reads, and no way to scale either. It is a legitimate choice for a low-volume control stream where a global sequence genuinely matters, and a capacity ceiling everywhere else.

Q: What happens to ordering when you add partitions to a live topic?

The key-to-partition mapping changes, so a key that used to land on partition 3 may now land on partition 7 while its older events are still unread on partition 3. Per-key order breaks across the repartition boundary. Safe approaches are over-provisioning partitions up front, or draining the topic before expanding, or making consumers version-conditional so the reorder is harmless.

Q: How is SQS FIFO ordering different?

Ordering is scoped to a message group id rather than a partition, and throughput is limited per group. A consumer holding an in-flight message for a group blocks later messages in that group until it is deleted or its visibility timeout expires — which means a single poison message stalls its whole group, a failure mode with no direct Kafka equivalent.

Q: Can I rely on message timestamps to restore order?

Only within a single producer with a monotonic clock. Across producers, clock skew of tens of milliseconds is normal and NTP corrections can move a clock backwards, so two events a few milliseconds apart may carry timestamps in the wrong order. Use a per-entity sequence number assigned by the writer of record instead.

Q: Does a dead-letter queue break ordering?

Yes, and that is usually the point. Parking a failing message lets the rest of the partition proceed, at the cost of the parked message being applied much later or not at all. If per-key order is a hard requirement, a failure has to stall the key rather than skip it — which is why DLQ policy and ordering policy have to be decided together.

Q: Is exactly-once needed for ordering?

They are separate properties. Deduplication stops an effect happening twice; ordering decides the sequence of distinct effects. A pipeline can be perfectly deduplicated and still apply `shipped` before `paid`, which is why realistic test suites attack both at once.

Brokers give total order only inside a partition, so ordering is something you buy with a partition key — and pay for in parallelism.

Written and reviewed by Sahil Srivastav

Distributed systemsMessagingDesign trade-off

What it actually is

An ordering guarantee is a promise about the sequence in which a consumer observes messages. The useful unit is almost never “globally”: Kafka orders messages within a partition, SQS FIFO within a message group, RabbitMQ within a single queue with a single consumer. Across those units there is no order at all, only the order in which things happen to arrive.

This is a deliberate consequence of how parallelism works. A totally ordered log is a single sequential resource, so total order across a topic means one writer, one consumer, and no horizontal scaling. Partitioning trades global order for throughput, and the partition key decides which subsets keep their order.

Consumers therefore need one of two properties. Either the messages that must be ordered land in the same partition — the key-based approach — or the handler is order-insensitive, which means it can apply events out of sequence and still converge. The second is often cheaper than engineers expect and scales far better.

Why it matters in production

Because out-of-order application corrupts state in ways that look like data-entry errors. An order that receives `shipped` before `paid` either rejects the valid transition or, worse, applies it and leaves a shipped-unpaid record. Support tickets from this class of bug are notoriously hard to trace, because by the time anyone looks, the events are long gone from the broker.

Ordering also silently disappears when the system is tuned for throughput. Raising a producer’s in-flight request limit without idempotence enabled lets a retried batch land after a later one; adding a thread pool inside a consumer lets two messages from the same partition be applied concurrently. Both are common performance changes with a correctness cost that only shows under load.

And ordering is the axis along which hot partitions appear. Keying by tenant gives per-tenant order and a partition that melts when one tenant is ten times the others — a trade-off worth naming out loud in a design round.

How it works

The partition key is the ordering contract

The broker hashes the key to a partition, so all messages sharing a key are appended to the same log in send order and consumed in that order by one consumer in the group. Choosing the key is therefore choosing the scope of your ordering guarantee: `order_id` orders each order independently and spreads load; `tenant_id` orders a whole tenant and risks a hot partition; no key round-robins and orders nothing.

How producers lose order

With more than one request in flight per connection and no idempotence, a failed batch can be retried after the batch behind it has already been appended, permanently inverting two messages in the log. In Kafka, `enable.idempotence=true` makes the broker enforce per-partition sequence numbers and reject out-of-sequence batches, which preserves order while keeping up to five requests in flight.

How consumers lose order

Anything that adds concurrency behind a single partition breaks its order: handing messages to an executor, batching then processing in parallel, or having two consumers read the same partition. If you need both parallelism and order, parallelise by key within the batch — group the messages by key and process each group sequentially — rather than by message.

Version-based convergence

The robust alternative is to make application order-insensitive. Each event carries a monotonic version or source timestamp for its entity, and the write is `UPDATE ... WHERE version < :incoming`, so an older event that arrives late matches no rows and is discarded. This survives redelivery, replay, and out-of-order arrival without any per-partition serialisation.

Buffering and reordering

When a handler genuinely cannot skip a step — a state machine with strict transitions — the consumer can park an event whose predecessor has not arrived and apply it after the gap closes. This is correct but expensive: it needs a keyed buffer, a timeout, and a decision about what to do when the missing event never arrives. Prefer it only where the state machine itself is the requirement.

Implementing it

Pick the narrowest key that satisfies the invariant. If the rule is “events for one order must apply in sequence”, key by order id — not by customer, and certainly not by tenant, both of which buy ordering you do not need at the cost of skew.

Turn on producer idempotence before touching in-flight limits or retry counts, otherwise the throughput change quietly reorders the log.

Carry a version or sequence number in the event payload and make every write conditional on it. Even in a strictly ordered pipeline this is worth doing: it makes replays and backfills safe, and it makes a future re-partitioning a configuration change rather than a rewrite.

Assert out-of-order behaviour in tests by shuffling the event sequence and checking final state, not just the happy-path order. If the suite only ever feeds events in order, the ordering assumption is untested.

Interview questions and how to answer them

What ordering does Kafka actually guarantee?

Total order within a partition, and nothing across partitions. Messages with the same key hash to the same partition, so per-key order holds provided the producer is idempotent and one consumer instance processes that partition sequentially. Consumers read partitions independently, so interleaving across partitions is arbitrary.

You need per-customer ordering and high throughput. How do you get both?

Key by customer id so each customer’s events share a partition, and scale by partition count rather than by threads per partition. If one customer is large enough to dominate a partition, either shard that customer’s key with a sub-key when its events are independent, or switch to version-conditional writes so ordering stops being required at all.

An event arrives whose predecessor has not. What should the consumer do?