ML engineering round
Data Leakage in a Feature Pipeline Coding Problem
Written and reviewed by Sahil Srivastav
What this interview round tests
Data leakage is the bug that makes a model look brilliant offline and fail the moment it is deployed. It happens when information from outside the training fold — from the test set, or from the label itself — sneaks into the features, so the model is quietly graded on data it should never have seen.
Interviewers probe this because it is the most common way ML work goes wrong, and it never raises an error. A scaler fitted on the full dataset before the split leaks test-set statistics. A feature derived from the target leaks the answer. The validation score goes up, which is exactly what makes it dangerous.
The scenario
You are given a feature pipeline and a train/test split. A preprocessing transform is fitted on the entire dataset before the split, and one feature is computed from the target column. The reported score looks strong — too strong.
The tests are structural and bounded, not exact: they assert the transforms are fitted on the training indices only, that the leaky feature is gone, and that the resulting test metric lands in a realistic band. Your job is to remove both sources of leakage so the evaluation is honest.
What you’ll practice
- Train/test discipline: fitting transforms on the training fold only
- Spotting target leakage from a feature derived off the label
- Why fit-then-split inflates offline metrics
- Structuring a pipeline so preprocessing cannot see the test set
- Reasoning about a metric that is suspiciously high
How to approach it
Leakage is information crossing a boundary it should not. Two boundaries matter here: the line between train and test, and the line between the features and the label. Trace where each transform learns its parameters from, and trace each feature back to its source — leakage hides wherever one of those boundaries is quietly crossed.
Because the tests are bounded rather than exact, aim for a pipeline that is structurally correct: fits confined to train indices, no target-derived features, a metric that is good but plausible rather than perfect.
The starter repository ships with a failing test suite and a bundled verify.sh. Reviewed reference solutions are part of Gronex Pro — this page stays spoiler-free on purpose.
FAQ
Do I need a GPU or a large dataset?
No. The problem is CPU-only and runs in seconds on a small dataset with scikit-learn. The skill being tested is train/test hygiene, not model training at scale.
Why are the tests bounded instead of checking an exact score?
Because exact metric values flake across library and platform versions. The tests assert the structural fix — fits on train only, no leaky feature — and that the metric lands in a correct band, which is what actually distinguishes a leaky pipeline from a clean one.