Data engineering round
Airflow DAG Dependency & Idempotent Load Coding Problem
Written and reviewed by Sahil Srivastav
What this interview round tests
Orchestration bugs are the ones that page a data team at 3am. A pipeline is a directed acyclic graph of tasks, and two invariants have to hold no matter what: every task runs only after the upstreams it reads from, and re-running the pipeline after a failure produces the same result as running it once. Break either and you get a table that is subtly, silently wrong.
Interviewers reach for this because it is the daily reality of the job. Schedulers retry. Backfills re-run. A load that appends instead of upserting turns every retry into a double-count, and a task that reads before its upstream has written picks up stale or empty data. This problem puts both failure modes in one small scheduler you can read end to end.
The scenario
You inherit a lightweight in-repo scheduler and a small set of tasks that extract, transform, and load a fact table. It works on a clean first run, but the bundled tests exercise what production actually does: they run the whole DAG twice.
On the second run the totals drift, and one task consumes data another task has not produced yet. Your job is to make the execution order respect the declared dependencies and make the load idempotent, so running the pipeline any number of times yields identical rows and correct totals.
What you’ll practice
- Topological execution: never run a task before the upstreams it depends on
- Idempotent loads: upsert semantics so a retry cannot double-count
- Reasoning about at-least-once execution and safe re-runs
- Separating declared dependencies from incidental ordering
- Data-quality assertions: row counts and totals that must balance across runs
How to approach it
Two invariants are in tension here — the order tasks execute in, and what a repeat run does — so reason about them separately. Ask what actually determines run order, and whether that matches the dependencies each task declares. Then ask what a second run does to state the first run already wrote, and which of the two the twice-run test is objecting to.
Treat the twice-run test as the specification. Anything that changes between the first and second run is a bug: either an ordering that let a task read too early, or a write that accumulated instead of converging.
The starter repository ships with a failing test suite and a bundled verify.sh. Reviewed reference solutions are part of Gronex Pro — this page stays spoiler-free on purpose.
FAQ
Do I need Airflow installed to try this?
No. The scheduler is a small self-contained Python module in the repo — the same dependency-and-idempotency concepts Airflow enforces, without the framework. It runs on CPU in seconds with pytest.
Why does idempotency matter so much in pipelines?
Because retries and backfills are normal, not exceptional. A pipeline that is only correct on a clean first run is a pipeline that corrupts data the first time something fails and re-runs. Idempotent loads are what make retries safe.