The Easiest Way to “Win” Is to Accidentally Cheat
Part 2.3: leakage, duplicate reports, evaluation roles, and the discipline behind an honest metric.
The easiest way to get a great machine-learning score is not always to build a great model.
Sometimes you accidentally give the model a clue about the answer.
That is data leakage: information from the evaluation set slips into training, directly or indirectly. The model then looks smart in a test that is no longer independent.
In an MRI project, leakage can be obvious—slices from the same examination appearing on both sides of a split. It can also be quiet: duplicated templates, repeated patients, site-specific shortcuts, or decisions made after repeatedly inspecting the same small evaluation set.
Phase 2 was where I made those boundaries explicit.
1. The basic rule: split by the thing you predict
The prediction unit here is a full MRI study, not an image slice.
Wrong: train on some slices from a study, validate on others
Right: every slice and series from one study stays together
If a validation image comes from the same examination as a training image, a model can learn the scanner appearance, anatomy, or incidental details of that study instead of learning the disease pattern.
This seems obvious once stated. It is still a common source of falsely optimistic results because image datasets are often stored as many files per patient or examination.
2. Why reports created a second leakage risk
The labels in this project are partly derived from reports. During review, I discovered that some normalized report text appeared in more than one study.
Identical report text does not prove identical MRI pixels or the same patient. But it is a warning sign: if the same report template appears in both training and validation, it can produce near-identical weak labels on both sides.
The provisional folds were therefore replaced with group-aware folds:
Normalize report wording
↓
Group matching reports together
↓
Assign the whole group to one fold
↓
Train and validate on separate groups
That is a practical example of a general principle: when you discover a possible dependency, group it before you trust the metric.
3. Separate training data from decision-making data
There are two different ways data can influence a project:
| Role | Allowed use |
|---|---|
| Weak training set | Fit the image model using report-derived labels and their masks/weights |
| Gold development set | Compare ideas, inspect errors, choose models; never train on it |
| Gold audit set | Final check only; never train on it or repeatedly tune against it |
The final audit set is intentionally boring. It stays out of the conversation until a candidate and evaluation code are frozen.
Why? If I repeatedly look at a small expert-labelled set while choosing architectures, thresholds, losses, and preprocessing, I am slowly training my own decisions on it. The code may never read those labels, but the project still overfits.
Interview concept — development-set overfitting: repeated human decisions can overfit a validation set just as model parameters can.
4. A validation plan has limits too
The supplied competition metadata does not provide reliable patient or hospital identifiers. That means I can group studies and report duplicates, but I cannot honestly claim full patient-level or site-level separation.
The right response is not to ignore that gap. It is to record it:
- report grouping is a useful leakage defense, not proof that all images are independent;
- protocol patterns can be balanced as proxies, not presented as hospital sites;
- full-image fingerprinting is still a future check when the whole image corpus is available;
- results on a small gold set need uncertainty and per-target reporting.
That last point is especially important. A tiny change in macro AUC may be noise when some findings have very few positive expert examples.
5. The safeguards, in one checklist
Before trusting the first image baseline, I want to be able to answer yes to each question:
- Are all slices from one study in exactly one split?
- Are normalized duplicate-report groups contained within a split?
- Are expert-labelled evaluation studies excluded from training?
- Is the final audit set untouched by iterative model choice?
- Do labels, folds, preprocessing, and code each have a version?
- Are results reported per target, not just as one flattering average?
- Are known limits written down rather than hidden behind a score?
This is not bureaucracy. It is the machinery that makes a future result interpretable.
What I learned
Leakage is not always a dramatic bug. Often it is a reasonable shortcut that quietly changes the question your metric answers.
The original question is: “Will this model generalize to a new MRI study?”
After leakage, the metric may be answering: “Can this model recognize something closely related to what it has already seen?”
Those are very different achievements.
The first baseline is next. It will probably be imperfect. That is okay. The important thing is that when it fails—or improves—we will have a better chance of knowing why.
Return to the Phase 2 overview →
This post shares aggregate validation practices only. It does not disclose competition images, reports, identifiers, case-level labels, private code, weights, or non-public leaderboard results. It is not medical advice or evidence of clinical validity.