Part III — Debugging Interactive and Numerical AI

The model bug that was never in the model

The notebook reproduces (Chapter 12): pinned env, seeded runs, hashed data, green Run All on two machines. And the fraud classifier is a star — 0.99 validation AUC overnight. The engineer begins tuning the architecture for the last point of precision. Then someone deploys it on last week’s live transactions: 0.61. Coin-flip with confidence.

# the star result (reproducible, pinned, and fictional)
val_auc = 0.99   # MEASUREMENT on the validation split
live_auc = 0.61  # MEASUREMENT on production week

OBSERVATION: same pinned notebook → 0.99 validation, 0.61 live; gap reproduces across reruns. HYPOTHESIS H1 (architecture deficit): the model underfits live patterns; capacity or tuning is the fix. HYPOTHESIS H2 (data defect): the validation number measures leakage or split corruption, not generalization — the model learned a shortcut that production does not offer. INFERENCE: none yet — validation AUC is a downstream symptom shared by both hypotheses. Only a data-vs-model swap separates them, and architecture tuning under H2 burns weeks improving a leak detector.

This chapter’s question: when validation shines and production fails, what ordered probe convicts the data before anyone touches the model?

Why tuning first fails

The obvious move — “try a bigger model / different LR” — fails because a leaked split rewards every architecture with fake signal. Three data defects masquerade as model bugs:

1. Label leakage. A feature computed from the target: days_to_chargeback in a fraud row, a refund_id prefix encoding the outcome (Chapter 9’s ghost, ML edition), a post-event timestamp smuggled into pre-event features. The model is honestly reporting that the future predicts the past.
2. Split corruption. Duplicates or near-duplicates across train/validation (same order_id in both), time-travel splits (train on December, validate on a November sample drawn from the same accounts), or preprocessing fitted on the full frame (scaler.fit(all_data) before splitting — validation statistics baked into training inputs).
3. Schema/distribution shifts. A column silently retyped (amount parsed as string in one dump), a category unseen in training dominating live traffic, a silent null-fill (fillna(0)) converting missingness into a confident zero. The loader from Chapter 8 has no contract, so the frame’s meaning drifts while its shape stays put.

OPINION: most “model bugs” that survive a week of tuning are data bugs that were never given a hearing. The architecture is the most expensive place to look and the least likely residence of a 0.99→0.61 gap.

These three categories are not this book’s invention. Kaufman and colleagues gave leakage a formal definition — “the introduction of information about the target that should not legitimately be available” — and named the two-part fix: rigorous data management plus a learn–predict separation that mirrors how the model will be used at inference (Kaufman et al., 2012). Kapoor and Narayanan later organized the same failures into an eight-type taxonomy: no clean train/test separation (missing split, preprocessing or feature selection on the full set, duplicates), using features not legitimately available at prediction time, and a test set that does not represent the population of interest (temporal leakage, train/test dependence, sampling bias) (Kapoor & Narayanan, 2023). The chapter’s H2 sub-cases are their taxonomy, restated for a debugging session.

The mental model: validation measures the split, not the model. A validation score is a claim about the relationship between two datasets wearing a model’s name. Debug the relationship (schema → distribution → leakage → split integrity) before debugging the function approximator.

The method: the train-vs-data swap probe

Ordered probe — schema before distribution before leakage before architecture, one swap per run:

1. Validate schema first. Assert dtypes, ranges, null rates, and category sets per split; diff train vs. validation vs. live on all four. A MEASUREMENT table, not a glance — amount.dtype, % null, top-5 categories, min/max per split.
2. Check split integrity. Duplicate-ID overlap across splits, time-ordering audit (max train timestamp < min validation timestamp for temporal problems), and a record of exactly where preprocessing was fitted. Any overlap or time-travel is a conviction of H2 before any model runs.
3. Run the discriminating swap: same-model × clean-data vs. same-data × baseline. Two single-variable runs: (a) hold the architecture fixed, retrain on a de-leaked, re-split dataset (suspect columns dropped, split rebuilt honestly) — prediction if H2: the 0.99 collapses toward live levels (~0.6), proving the score was leak-bought; (b) hold the suspect data fixed, train a trivial baseline (logistic regression / majority-class) — prediction if H2: the baseline also scores ~0.99, proving the signal is in the leakage, not the architecture. If (a) stays high and (b) scores near chance, H2 is exonerated and H1 (model) earns its tuning budget.
4. Treat correlation as suspect, never as diagnosis. A feature-importance chart naming days_to_chargeback is not a leakage conviction — it is a lead. The conviction is the swap result.

    flowchart TD
    S["profile schema per split: dtype, null rate, ranges, category sets"] --> SD{"train / val / live agree?"}
    SD -->|no| H2s["H2: schema / distribution shift — fix the loader contract"]
    SD -->|yes| I["split integrity: duplicate-ID overlap, timestamp ordering, where preprocessing was fit"]
    I --> ID{"overlap or time-travel found?"}
    ID -->|yes| H2l["H2: leakage / split corruption — rebuild the split"]
    ID -->|no| SW["run both swap arms"]
    SW --> A["Run A: same model x de-leaked, honestly re-split data"]
    SW --> B["Run B: trivial baseline x suspect data"]
    A --> V{"A collapses toward live AND B still ~0.99?"}
    V -->|yes| H2["H2 convicted: the score was leak-bought — quarantine + regression test"]
    V -->|"A holds high, B near chance"| H1["H1: data exonerated — the model earns its tuning budget"]
  

# leakage swap probe (reader executes; predictions pre-written below)
SUSPECT_COLS = ["days_to_chargeback", "refund_id_prefix"]  # named suspects, not vibes
# Run A (same model x clean data): drop suspects, rebuild split honestly, retrain identically
train_clean, val_clean = rebuild_split(df.drop(columns=SUSPECT_COLS), time_col="event_ts")
auc_clean = train_and_eval(model_fn, train_clean, val_clean, seed=SEED)
# Run B (same data x baseline): trivial model on suspect data
auc_baseline = train_and_eval(logistic_baseline, train_suspect, val_suspect, seed=SEED)
print(f"suspect-model={0.99} clean-model={auc_clean:.3f} suspect-baseline={auc_baseline:.3f}")
# Prediction H2 (leakage): auc_clean collapses toward live (~0.6); auc_baseline stays ~0.99.
# Prediction H1 (model deficit): auc_clean stays high; auc_baseline near chance.

OBSERVATION (constructed illustration, not a measured run): duplicate audit found 18% of validation order_ids in train; Run A collapsed 0.99 → 0.63; Run B (logistic) scored 0.97 on suspect data. UPDATED BELIEF: H2 supported — split overlap plus a leaky post-event column; architecture tuning suspended. INFERENCE: the fix is a split-rebuild + column quarantine with a leakage regression test, not a larger network. The tuning budget was never owed.

Research lineage: leakage is common, and it is the reproducibility crisis

The “trivial model scores suspiciously well” probe is established practice. Kaufman and colleagues list, among their detection methods for when you did not control data collection, exactly this move: fit an over-simple model and see whether it already achieves near-perfect performance — if it does, the signal is in the leak, not the hypothesis space (Kaufman et al., 2012). This chapter’s Run B is that probe with a written prediction attached.

Leakage is not a rare mistake. Kapoor and Narayanan surveyed seventeen scientific fields that had adopted machine learning and found leakage-driven errors in 294 papers, in several cases invalidating the field’s headline results; they argue leakage is the leading mechanism behind machine-learning’s reproducibility problems (Kapoor & Narayanan, 2023). Their proposed “model info sheets” matured into REFORMS, a 32-item consensus reporting checklist — built by nineteen researchers across computer science, statistics, the social sciences, and biomedicine — that forces the split, the features’ availability at prediction time, and the population of interest to be stated explicitly (Kapoor et al., 2024). Chapter 12’s reproducible-but-wrong notebook and this chapter’s leak-bought 0.99 are the same failure seen from two angles.

And the audit can be partly automated. Yang and colleagues built a static data-flow analysis that flags three leakage patterns directly in notebook code — overlap (the same rows in train and test), multi-test (a test set consulted more than once during development), and preprocessing leakage (fit called before the split) — and ran it across more than 100,000 public notebooks, finding at least one of the three pervasive (Yang et al., 2022). The schema-and-split audit step of this chapter is the manual form; where a linter like theirs is in the stack, run it first and treat its flags as leads for the swap, not as convictions.

Schema validation is a solved engineering problem. Breck and colleagues describe the data-validation system Google runs in front of production model training: a declared schema per feature, automated anomaly detection against it, and explicit training/serving skew checks that compare the training distribution to what the deployed model actually receives (Breck et al., 2019). The 0.99-vs-0.61 gap is training/serving skew; the schema-first audit step is a hand-run version of their system.

And a clean split can still lie if the labels are wrong. Northcutt, Athalye, and Mueller estimated label errors across ten commonly used benchmark test sets — averaging about 3.4%, and higher in some (roughly 6% of the ImageNet validation set, over 10% of QuickDraw) — and showed that correcting them can reorder model leaderboards, so that a lower-capacity model sometimes overtakes a higher-capacity one once the test set is right (Northcutt, Athalye & Mueller, 2021). A mislabeled validation set measures neither the split nor the model. Where the swap arms above both come back inconclusive, sample and hand-check the validation labels before extending the tuning budget.

Lab 13: train-vs-data swap probe

PROPOSED, not executed: no author-measured results are reported. The evidence this chapter requires is the reader’s own swap table.

Setup. Take any suspiciously good validation score (or inject leakage: copy 15% of train rows into validation with new indices, or add label_copy = y + tiny_noise as a feature). Freeze the architecture, seed, and training command — they are controlled variables throughout.

Task.

1. Write H1 (model deficit) and H2 (data defect: leakage/split/schema) with distinct swap predictions before retraining — numeric FORECASTs with tolerances (e.g., “Run A drops ≥0.2 if H2”).
2. Audit schema + split integrity first and record the MEASUREMENT table (overlap %, timestamp ordering, null/dtype diffs). Independent variable: data honesty (suspect vs. cleaned) / model capacity (full vs. baseline); controlled variables: seed, architecture per run arm, training budget.
3. Execute Run A (same model × clean data) and Run B (same data × baseline), each with ≥2 repeated trials (seeds 42/43) since training is stochastic — report means, not singles.
4. Record OBSERVATION per arm and UPDATED BELIEF. Conviction requires both arms to agree (collapse + baseline-high); a split verdict is UNKNOWN with the next audit named.

Success criterion. A completed swap table with both arms agreeing, plus the quarantine artifact (dropped-column list + split-rebuild script + regression assertion). A retrained model without the audit table is explicitly not completion — the table is the diagnosis.

Companion tool: Data Validation Inspector

What it accepts: per-split schema profiles (dtypes, ranges, null rates, category sets), the ID-overlap and timestamp-ordering audit, the suspect-column list, and the two-arm swap table. What it performs: it enforces audit order (schema → split integrity → swap), refuses a data-health verdict while any audit cell is UNKNOWN, checks swap predictions against pre-written FORECASTs, and records the quarantine (columns dropped, split script hash). What it can establish: whether the validation score measures the split rather than the model — and which defect class (leakage / overlap / time-travel / schema shift) the audit convicts, under the examined datasets only. What it cannot establish: model correctness, optimal architecture, or production performance — clean data can still train a bad model (Chapters 14–15), and today’s split rots with tomorrow’s distribution. It never treats feature importance, correlation, or a single high score as diagnosis. How its output changes your next action: an H2 conviction routes to split-rebuild + column quarantine + loader contracts (Chapter 8 guards at the data handoff); an H1 exoneration-with-clean-data routes to Chapters 14–15 with the data certified in writing as the training input.

Paper form, sufficient for this chapter:

Splits: train ___ rows / val ___ rows / live ___ rows (hashes: ___)
ID overlap train∩val: ___%   Time ordering OK? Y/N (max-train ___ < min-val ___?)
Schema diffs (dtype/null/category per column): ___
Suspect cols: ___   FORECAST A (collapse ≥ ___): ___   FORECAST B (baseline ≥ ___): ___
Run A (same-model×clean): ___ / ___   Run B (same-data×baseline): ___ / ___
CONVICTION: H1 model / H2 data (circle; defect class: ___)
QUARANTINE: dropped ___ | split script hash ___ | regression assertion ___

Where a software implementation does not yet exist in the reader’s stack, this record is the tool. The data-first discipline precedes any automation.

Reusable procedure: every suspicious score gets this before tuning

1. Freeze the scoreboard — record suspect validation + any live number with hashes.
2. Audit schema and splits — profiles per split, overlap %, timestamp ordering.
3. Name suspect columns explicitly — no “probably leakage somewhere.”
4. Swap both arms with pre-written numeric forecasts and repeated trials.
5. Quarantine and contract — drop/rebuild/guard, regression test at the loader.

Failure modes

• Tuning-first reflex. Burning GPU weeks on architecture while the split overlaps 18%. Tuning a leak detector makes a better leak detector.
• Importance-as-conviction. “SHAP says the column matters, so it’s leakage.” Importance measures reliance, not legitimacy — the swap is the verdict.
• Fit-on-everything preprocessing. Scaling/imputing/encoding before splitting. Validation information enters training through the back door; the score is compromised before the first epoch.
• Time-travel splits. Random splits on temporal data. The future trains the past; production, which only has the past, disagrees.
• Single-seed swap. One retrain per arm treated as proof. Training is stochastic — arm means over ≥2 seeds or the verdict is UNKNOWN.
• Correlation-as-diagnosis. A 0.9 correlation between a feature and the label presented as a leakage finding. Correlation is a lead for the audit, never the conviction.

Limits, per contract: one swap record convicts one data defect under one dataset revision; it does not certify the model, the evaluation (Chapter 16), or future distributions. UNKNOWN where audit cells are unmeasured or trials are single.

References

• Shachar Kaufman, Saharon Rosset, Claudia Perlich, and Ori Stitelman. Leakage in Data Mining: Formulation, Detection, and Avoidance. ACM Transactions on Knowledge Discovery from Data 6(4), 2012, article 15. https://doi.org/10.1145/2382577.2382579
• Sayash Kapoor and Arvind Narayanan. Leakage and the Reproducibility Crisis in Machine-Learning-Based Science. Patterns 4(9), 2023, 100804. https://doi.org/10.1016/j.patter.2023.100804
• Eric Breck, Neoklis Polyzotis, Sudip Roy, Steven Euijong Whang, and Martin Zinkevich. Data Validation for Machine Learning. Proceedings of Machine Learning and Systems (MLSys), 2019. https://mlsys.org/Conferences/2019/doc/2019/167.pdf
• Curtis G. Northcutt, Anish Athalye, and Jonas Mueller. Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks. NeurIPS Datasets and Benchmarks Track, 2021. https://arxiv.org/abs/2103.14749
• Sayash Kapoor, Emily M. Cantrell, Kenny Peng, Thanh Hien Pham, Christopher A. Bail, Odd Erik Gundersen, Jake M. Hofman, Jessica Hullman, Michael A. Lones, Momin M. Malik, et al. REFORMS: Consensus-Based Recommendations for Machine-Learning-Based Science. Science Advances 10(18), 2024, eadk3452. https://doi.org/10.1126/sciadv.adk3452
• Chenyang Yang, Rachel A. Brower-Sinning, Grace A. Lewis, and Christian Kästner. Data Leakage in Notebooks: Static Detection and Better Processes. Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering (ASE), 2022. https://doi.org/10.1145/3551349.3556918

Debugging Checklist

• Suspect validation + live numbers recorded with data hashes?
• Per-split schema profiles + overlap % + timestamp ordering measured?
• Suspect columns named explicitly before any retraining?
• H1/H2 with numeric collapse forecasts written before the swap?
• Both swap arms run with ≥2 seeds; means reported?
• Quarantine artifact (dropped cols + split script + regression assertion) committed?
• No importance/correlation/single-score treated as conviction?

What This Chapter Established

• The data-first rule: validation scores measure the split-model relationship; schema → split-integrity → swap is the ordered probe, architecture last. Leakage here is a property of the data and its split — the training input. Chapter 16 meets leakage again as a property of the evaluation instrument: the same diagnostic concept pointed at a different debugging object, not a repeat of this chapter.
• The train-vs-data swap (same-model×clean-data + same-data×baseline) with pre-written forecasts and repeated trials, demonstrated on the 0.99→0.61 fraud case via 18% overlap and a post-event column — constructed illustration, no measured runs claimed. The “trivial model already wins” arm is a named leakage-detection method (Kaufman et al.).
• Empirical grounding: leakage has a formal definition and an 8-type taxonomy (Kaufman et al.; Kapoor & Narayanan), it affects 294 papers across 17 fields and drives ML’s reproducibility crisis, the proposed fix has matured into a 32-item consensus checklist (REFORMS, Kapoor et al. 2024), overlap/multi-test/preprocessing leakage is statically detectable and pervasive across 100k+ public notebooks (Yang et al. 2022), and schema/skew validation is a solved engineering problem (Breck et al.). The val-vs-live gap is training/serving skew. A clean split still misleads when the labels are wrong: ~3.4% mean label errors across ten benchmark test sets, enough to reorder leaderboards (Northcutt et al. 2021).
• Lab 13 as a proposed swap record the reader executes; the Data Validation Inspector contract (accepts/performs/can-establish/cannot-establish/next-action).
• What was NOT proved: model correctness under clean data, evaluation validity, or robustness to future distribution shift.
• Forward link: the data is now honest — and training still crashes on the first batch with a shape error no audit table predicted. Clean contents, wrong containers. That mismatch is next.

Next

The split is honest, the columns are legitimate, the score is real — and matmul disagrees about what a batch looks like. Features leave the loader as one shape, arrive at the layer as another, and the error message blames broadcasting three frames away from the actual handoff. The next chapter systematizes the most common numerical failure in AI code: shapes, types, and devices.