Before Training a Model, I Had to Build a Data Factory

Part 1 of this series was about the beginning of a machine-learning project: define the problem, inspect the data, design validation, and prove the submission plumbing.

Phase 2 was about making the inputs trustworthy enough to train on. Not about chasing a bigger model.

For this knee-MRI competition, that meant solving three connected problems:

MRI files          → can we turn them into consistent model inputs?
Radiology reports  → can they become cautious training signals?
Validation         → can we tell a real improvement from an accidental shortcut?

This is the short version. Each decision below links to a deeper post for readers who want the engineering detail.

The outcome in plain English

By the end of Phase 2, I had a repeatable path from messy source material to a training-ready research dataset:

1. MRI series are physically ordered, checked for geometry problems, and converted into a consistent model-ready representation.
2. Multilingual radiology reports are converted into weak labels—useful training hints, not unquestionable truth.
3. Missing or disputed findings are masked instead of quietly turned into “no disease.”
4. Gold expert-labelled cases are held out from training, and near-duplicate report groups are kept from leaking across folds.

The headline number is 34,577 usable report-derived targets across 12 findings. That is more supervision than the 58 fully expert-labelled studies alone, but it comes with weights, masks, provenance, and explicit uncertainty.

That distinction matters. More labels are not automatically better labels.

The job was not to manufacture certainty. It was to preserve uncertainty in a form a model can learn from.

The traditional lifecycle, revisited

In Part 1, I described machine learning as a loop rather than “train, tune, deploy.” Phase 2 filled the middle of that loop:

The order matters. Training before these choices are settled creates a fast way to optimize noise.

Three deep dives

1. MRI is a geometry problem before it is a vision problem

An MRI examination is made of series, and series are made of slices. File names are not a trustworthy sequence. The useful order comes from physical position metadata.

I also learned that a technically valid series can still be a poor training input: localizer-like scans, broad views, and edge-of-volume images are not broken files, but they can dilute the signal.

Read: An MRI scan is not a stack of images →

2. Reports can teach, but they cannot be treated as ground truth

The reports are multilingual and written for clinicians, not for a classifier. “Not mentioned” is not the same as “absent.” A phrase can be uncertain, historical, negated, or anatomically ambiguous.

The pipeline therefore combines rules with bounded model review, keeps evidence provenance, and masks cases where the evidence is too weak. The result is a more useful training set without pretending a language model is a radiologist.

Read: Can radiology reports teach an MRI model? →

3. A high validation score can be a bookkeeping error

One of the most valuable Phase 2 findings was not a model result. It was a leakage risk: identical or near-identical report text could be split across provisional folds.

That is dangerous because a model may appear to generalize when the evaluation set contains a close cousin of something it already saw. The answer was to freeze evaluation roles, keep gold data out of training, and group normalized duplicate reports together.

Read: The easiest way to “win” is to accidentally cheat →

What changed because of the evidence

The best Phase 2 decisions were behavioral changes, not charts or model names:

• I stopped relying on filename order and made physical slice position mandatory.
• I rejected a tempting hard image-content filter after visual review showed it would discard useful diagnostic series.
• I kept DICOM as the source of truth while using reduced memory-mappable arrays for faster training reads.
• I refused to convert silence in a report into a negative label.
• I replaced provisional folds after finding duplicate-report leakage risk.
• I kept model disagreements low-weight or masked instead of “picking a winner” everywhere.

This is what useful exploratory work should do: change the next action.

What Phase 2 does not prove

It does not prove clinical accuracy. It does not prove that label v2 will beat label v1. It does not prove the final image model will perform well.

It proves something narrower and still important: the next baseline will start with tested image inputs, explicit label uncertainty, and a validation boundary designed to resist easy self-deception.

The next phase is deliberately boring in the best way: implement exact metrics, train a small baseline, compare label versions under the same conditions, and learn from the failures.

Series map

Part 1  Foundation: problem, profiling, validation, submission plumbing
   ↓
Part 2  Phase 2 overview: trustworthy inputs before training
   ├── Part 2.1  MRI data pipeline
   ├── Part 2.2  Weak supervision from reports
   └── Part 2.3  Leakage-resistant validation
   ↓
Part 3  First image baseline (after evidence exists)

This is a competition research prototype, not a clinical diagnostic system. The post shares aggregate engineering lessons only; it does not include competition images, report text, identifiers, private code, weights, or case-level findings.