I Entered a Kaggle Competition to Learn Model Training

I have built AI products, worked with language models, and shipped systems around models. But there is a difference between using a model and becoming good at training one.

I wanted to close that gap. The obvious route was another course. The more useful route, I decided, was to enter a competition where the data is messy, the metric is unforgiving, compute is limited, and every shortcut eventually shows up in validation.

So I entered the 2026 RSNA Knee Abnormality Detection competition on Kaggle.

This series is my learning log. It is written for two audiences: anyone learning how real machine-learning projects work, and my future self revising for an interview. I will explain the concepts, record the decisions, and separate what the evidence says from what I merely hope will work.

The first lesson arrived before I trained anything: model training is only one stage of a much larger machine-learning system.

---

The competition in plain English

Each example is a knee MRI examination. An examination is not one picture. It contains multiple series—groups of image slices captured from different directions and with different scanner settings.

The task is to predict the probability of twelve findings, including ligament and meniscus injuries, osteoarthritis, fluid-related findings, bone contusion, and fracture.

One knee examination
        ↓
Several MRI series
        ↓
Dozens or hundreds of image slices
        ↓
12 probabilities between 0 and 1

The main score is macro-average ROC AUC. In plain English, the model must rank positive cases above negative cases for each finding, and every finding receives equal weight in the final average. Performing brilliantly on a common finding does not compensate for ignoring a rarer one.

There is also an efficiency track, so speed matters alongside accuracy. Submissions run as offline Kaggle notebooks under a time limit. That makes file loading, preprocessing, memory use, and reproducibility part of the problem—not afterthoughts.

The unusual part is supervision. The training set contains 4,407 studies and radiology reports, but only 58 studies have complete expert labels for all twelve targets. The test set has images but no reports.

That creates an interesting learning problem:

Training time: images + reports + a small expert-labelled set
Test time:     images only

My high-level strategy is therefore reports teach; images infer. Reports can help create cautious training signals, but the final prediction system must work from MRI images alone.

This is a research prototype for a competition, not a clinical diagnostic system.

The traditional machine-learning lifecycle

People often compress machine learning into three words: train, tune, deploy. A more realistic lifecycle looks like this:

1. Frame the problem and metric
          ↓
2. Obtain data and understand the rules
          ↓
3. Profile the data
          ↓
4. Clean, sample, and define preprocessing
          ↓
5. Design validation and prevent leakage
          ↓
6. Build the simplest end-to-end baseline
          ↓
7. Train and compare models
          ↓
8. Perform error analysis
          ↓
9. Optimize accuracy, latency, and cost
          ↓
10. Reproduce, submit, deploy, and monitor

It is not perfectly linear. New evidence sends you backward. A validation problem can force a new split. A slow loader can force a new storage format. Error analysis can reveal that the label definition was wrong.

That loop is the real work.

Where we are in that lifecycle

The important word in that table is deliberately. Not training yet is not lack of progress. It is risk reduction.

Stage 1: problem framing before code

An interview question such as “How would you build a model for this?” is rarely asking for a model name first. A strong answer begins by defining:

• the prediction unit: one MRI study, not one image slice;
• the outputs: twelve probabilities, so this is multilabel classification;
• the metric: macro ROC AUC;
• the available information at training and inference time;
• the operational constraints: offline execution, runtime, memory, and privacy;
• the cost of mistakes and how results will be validated.

Those choices shape everything downstream. Because macro-AUC weights each target equally, I will need per-target metrics. Because the prediction unit is a study, slices from one study must never be scattered across training and validation. Because reports disappear at test time, a text-only model cannot be the final system.

What I learned: the metric and inference contract are part of the architecture.

Stage 2: data profiling is decision-making, not chart-making

Data profiling—or exploratory data analysis—is the process of understanding shape, quality, coverage, and anomalies before modeling.

I first downloaded only the small metadata files. That was enough to establish:

• 4,407 training studies;
• multiple MRI series per study;
• three anatomical planes: sagittal, coronal, and axial;
• large variation in series count and scan configuration;
• only 58 completely expert-labelled studies;
• reports that can provide useful but imperfect supervision.

The key output of profiling was not a dashboard. It was a set of decisions:

1. Do not download roughly 570 GB of images to a workstation without a reason.
2. Use Kaggle’s mounted data for image work.
3. Treat reports as weak supervision rather than ground truth.
4. Validate at study level.
5. Inspect a representative sample before processing everything.

Interview concept — weak supervision: training signals that are useful but less reliable than carefully reviewed ground truth. Weak labels should carry uncertainty; blanks must not silently become negatives.

What I learned: useful EDA ends with changed behavior. If a chart does not affect sampling, validation, preprocessing, or modeling, it may be decoration.

Stage 3: representative sampling before scale

Auditing every scan would be expensive. Auditing a random handful could miss rare but important configurations. I therefore created a deterministic sample of 160 studies:

• all 58 expert-labelled studies;
• 102 additional studies selected across series counts, fluid-sensitive proportions, repeated-view patterns, and report-length groups.

“Deterministic” means the same seed and inputs produce the same selection. That matters because another person—or future me—must be able to reproduce the audit.

This is an example of stratified sampling: sampling deliberately across meaningful subgroups instead of hoping uniform randomness covers them.

What I learned: the purpose of a sample is not to be small. It is to preserve the variation most likely to break the system.

Stage 4: test the data loader before the model

MRI files use the DICOM format. A DICOM file contains pixels plus metadata such as orientation, physical spacing, scanner information, and series identity.

The private audit examined:

• 160 studies found out of 160 requested;
• 983 image series;
• 34,738 individual files;
• 2,949 sampled pixel decodes;
• zero sampled decode failures;
• zero mismatches between supplied plane labels and DICOM orientation;
• regular slice-position spacing throughout the audited sample.

That is encouraging, but “the file opens” is not the same as “the image is useful.” A targeted visual review found broad-field, edge-of-volume, and localizer-like series mixed with diagnostic knee series. These are valid files that may still be poor training inputs.

The images also vary widely in matrix size, slice count, scanner, field strength, and physical spacing. A production loader cannot assume every scan is 512 by 512 or that every series has the same number of slices.

Interview concept — data validation: schema checks ask whether a value has the right type; semantic checks ask whether it means what the model expects. A perfectly readable localizer scan can still be semantically wrong for the task.

What I learned: the data loader is part of the model. Incorrect ordering, distorted resizing, or poor series selection can destroy signal before the neural network sees it.

Stage 5: validation is where honest machine learning begins

A model is useful only if its validation score predicts performance on unseen data. That requires separation between what the model learns from and what evaluates it.

I created a provisional five-fold split across all 4,407 studies. Each fold contains about 881 studies and 11 or 12 of the 58 expert-labelled studies. Every target has at least one positive expert example in every fold, and scan-configuration groups are spread across the folds.

Why “provisional”? The supplied metadata does not provide patient or hospital identifiers. I can guarantee that one study is assigned to one fold, but I cannot yet prove that different studies from the same patient—or unusually similar scans from one site—never cross folds.

That distinction matters because of data leakage: information from validation accidentally influences training, making the score look better than real-world performance.

Bad split: slices from one study appear in both train and validation
Result:    model recognizes the examination instead of generalizing

Better:    keep every study intact inside one fold
Best here: also group by patient and site—if reliable identifiers become available

With only 58 gold cases, the validation score will also have high uncertainty. I will report per-target results and variability, not hide everything behind one macro number.

What I learned: validation is not a final test you bolt on later. It is the contract that determines which experiments you are allowed to believe.

Stage 6: a baseline does not need to be intelligent

Before training a model, I built an offline notebook that predicts 0.5 for every finding.

It is intentionally useless as a diagnostic model. It was extremely useful as a system test.

The notebook checked:

• exact output column names and order;
• exact study identities and row order;
• no duplicated studies;
• finite probabilities between zero and one;
• offline execution without internet access;
• the competition’s notebook-backed submission route.

The first notebook version failed because Kaggle mounted a versioned dataset under a different directory layout than expected. I fixed discovery of both layouts, reran it, and completed a valid submission.

That failure was cheap because no training was involved.

Interview concept — baseline: the simplest repeatable system that establishes a reference. A baseline can test model quality, but it can also test integration. Always be clear about which one it proves.

What I learned: prove the whole path with a deliberately simple component before inserting an expensive one.

What comes next: storage, preprocessing, then the first model

The immediate experiment compares three ways to feed images into training:

I will measure storage size, conversion time, read throughput, peak memory, and whether the pipeline can keep a GPU busy. The point is not to crown a universally best format. It is to choose the best format for this dataset and compute budget.

Then the first real image pipeline will look roughly like this:

Raw MRI study
      ↓
Choose useful series
      ↓
Order slices using physical geometry
      ↓
Normalize intensity and spatial shape
      ↓
Sample a consistent slice budget
      ↓
Train a small baseline model
      ↓
Evaluate by fold, finding, runtime, and failure mode

Only after that baseline is trustworthy will I compare more sophisticated encoders or multi-plane aggregation methods.

My approach to complex machine-learning problems

This project is already giving me a reusable framework:

1. Write the contract. Define the unit of prediction, labels, metric, inference inputs, constraints, and non-goals.
2. Profile before scaling. Learn the dataset’s real shape and convert observations into decisions.
3. Sample for failure modes. Preserve important variation, not merely average cases.
4. Design validation before tuning. Decide what evidence will count before seeing model results.
5. Prove the thin vertical slice. Run one example through input, validation, output, and submission.
6. Measure bottlenecks. Benchmark data, compute, memory, latency, and cost rather than guessing.
7. Start with a small baseline. Complexity must earn its place through repeatable improvement.
8. Perform error analysis. Ask where and why the system fails, not only what score it achieved.
9. Keep provenance. Version data, splits, labels, code, and experiments so results can be reproduced.
10. State uncertainty honestly. “Provisional,” “sampled,” and “not yet tested” are useful engineering terms.

That is also how I would structure an interview answer. Start with the problem and evidence, walk through the lifecycle, name the risks, and make every technology choice serve a measured need.

The series from here

This is Part 1. Planned posts will follow the evidence rather than a fixed publication calendar:

1. Foundation: problem framing, profiling, validation, and the first submission.
2. Building the MRI data pipeline: DICOM, geometry, series selection, storage, and preprocessing.
3. Learning from reports: weak supervision, multilingual text, uncertainty, and label quality.
4. Training the first baseline: architecture, loss, metrics, experiment tracking, and error analysis.
5. Improving without fooling myself: ablations, transfer learning, runtime, and validation uncertainty.
6. Retrospective: what worked, what failed, and what I would change—after the competition and publication rules permit it.

I entered to get better at training models. So far, the competition has taught me something more fundamental: good model training begins by building a process that makes bad assumptions visible early.

---

Official references

• Competition overview
• Competition data description
• Evaluation
• Official rules
• RSNA challenge announcement

This post shares aggregate engineering lessons only. It does not publish competition images, reports, identifiers, private code, or case-level findings. Nothing here is medical advice or evidence of clinical validity.