Data engineering round
Incremental CDC Merge & Late-arriving Data Coding Problem
Written and reviewed by Sahil Srivastav
What this interview round tests
Full reloads do not scale, so real pipelines run incrementally: process only what changed since last time, and merge it into the target. That efficiency comes with two traps — records that arrive late, after the watermark has already moved past their event time, and merges that keep the wrong version of a key when updates come out of order.
Interviewers use this because incremental correctness is where data engineering gets genuinely hard. A watermark that is too aggressive silently drops late data. A merge that dedupes by key without ordering keeps whichever row it saw last, which may be stale. Both produce a target table that looks complete and is not.
The scenario
You are given an incremental loader that reads a batch of change records and merges them into a target keyed by a natural id. The tests feed it late-arriving records and multiple updates to the same key in a non-obvious order.
Late records get dropped, and some keys end up holding an older version than they should. Your job is to fix the watermark handling so late data is not silently lost and the merge so each key converges to its newest version regardless of arrival order.
What you’ll practice
- Incremental loads: watermarks and what "since last run" really means
- Handling late-arriving data without dropping it
- CDC merge semantics: newest-version-wins per key
- Deduplicating change records by key with correct ordering
- Verifying convergence: the target reflects the true latest state
How to approach it
Separate two concerns that are easy to conflate: which records a run selects, and how those records merge into the target. Late data is a selection question — does the run see a record whose event time predates the watermark but that arrived late? Keeping the right version per key is a merge question. Ask which of the two each failing test is pointing at.
Let the test fixtures — late records and shuffled updates — define correctness: the final target must match the true latest state of every key.
The starter repository ships with a failing test suite and a bundled verify.sh. Reviewed reference solutions are part of Gronex Pro — this page stays spoiler-free on purpose.
FAQ
What is late-arriving data?
A record whose event happened before your current watermark but that only reaches the pipeline now — a delayed upload, a retried export, a source that batches. A naive incremental filter excludes it, so it is silently lost.
Why not just full-reload every time?
Full reloads are correct but do not scale to large tables or frequent runs. Incremental loads are the norm in production, which is exactly why getting their edge cases right is a common interview signal.