Part III — Debugging Interactive and Numerical AI

The loss curve that says nothing — three times

Tensors flow (Chapter 14): every handoff green, batches well-formed, loss computes. And the curve is dead:

epoch 1: loss=2.303  epoch 5: loss=2.302  epoch 20: loss=2.302

Flat at ln(10) ≈ 2.303 — chance-level for ten classes. The team has seen this before: “learning rate too low, crank it.” They crank it. Now:

epoch 1: loss=2.301  epoch 2: loss=nan

Three candidate explanations circulate: the optimizer step is wrong (LR/pathology), the data teaches nothing (labels shuffled or inputs normalized to constants), or nothing is actually wrong — the logged number is not the optimized number (logging averages the wrong tensor, eval runs on an unaugmented copy, the curve plots a stale buffer).

OBSERVATION: loss flat at chance for 20 epochs; LR increase converts flat to NaN; training and logging code paths unexamined. HYPOTHESIS H1 (optimization pathology): LR schedule, gradient flow, or loss construction prevents descent (LR too low/high, gradients zeroed by a bug, wrong reduction averaging padding). HYPOTHESIS H2 (data pathology): the batches carry no learnable signal (shuffled labels, constant features, augmentation destroying content) — descent is impossible, not merely slow. HYPOTHESIS H3 (instrumentation pathology): training descends (or diverges) but the reported curve does not show it — logging/eval bug, not a training failure at all. INFERENCE: none yet — one flat curve is a downstream symptom of all three. Only single-variable interventions with pre-written curve predictions separate them.

This chapter’s question: when the curve misbehaves, what ordered triage convicts optimizer, data, or instrumentation — without changing all three at once?

Why “tune the LR” fails first

The obvious move — sweeping learning rates — fails because it assumes H1 while H2/H3 are live, and because LR changes mask the other two. The failure taxonomy before any sweep:

1. H1 signatures (optimizer/loss): loss flat with healthy gradients flowing is not H1 — check first. True H1 shows zero/collapsed gradients (grad.norm ≈ 0), exploded gradients pre-NaN, LR of 0 via scheduler misconfiguration (scheduler.step() per batch vs. per epoch collapsing LR to ~0 by epoch 2), a loss that cannot descend by construction (reduction over padding, double-softmax, target dtype silently cast), or — a distinct NaN cause from LR-too-high — half-precision overflow: float16 has a narrow dynamic range, so gradients or activations can hit inf/nan even at a reasonable learning rate. The fix is loss scaling (GradScaler / AMP) or bfloat16, not an LR change; the tell is that the NaN appears only with mixed precision enabled and the pre-NaN gradient norm is large but not diverging across steps. This is Chapter 14’s dtype-telepathy failure mode arriving during training rather than at the first batch.
2. H2 signatures (data): batches whose labels are decorrelated from inputs. The one-epoch overfit test is decisive (below): a healthy pipeline memorizes a 64-sample subset to ~0 loss; failure to overfit convicts the data path — or the model path — but never the LR schedule.
3. H3 signatures (instrumentation): the oldest lie in training. loss.item() logged before backward from a detached copy; epoch average dividing by wrong batch count; eval-mode curve plotted while train-mode loss descends; TensorBoard smoothing hiding divergence; checkpointed weights improving while the printed number stalls. The weights are the training; the curve is journalism about the training.

OPINION: half of all “training failures” brought to senior engineers are H3. Check what number you are looking at before changing what the number measures.

Two of the checks in this method are canon. Verifying that the loss at initialization equals ln(num_classes) for balanced classification — the 2.303 the opening curve is stuck at — and overfitting a single batch to near-zero are both in Karpathy’s widely-cited A Recipe for Training Neural Networks, alongside “fix the random seed” and “visualize just before the net” (Karpathy, 2019). This chapter’s contribution is not the probes; it is running them in exoneration order with written curve predictions instead of as a checklist.

The mental model: training is three coupled systems — the optimizer loop, the data stream, the instrumentation — and the curve is the output of the third describing the first fed by the second. Triage order is H3 → H2 → H1: verify the journalism, then the food supply, then the engine. Reversing the order burns GPU on fiction.

The method: one-variable training triage

Ordered probe, cheapest and most exonerating first, predictions pre-written per intervention:

1. Interrogate the instrumentation (H3) with zero training cost. Log raw per-batch loss alongside the epoch average for 50 steps; compare the plotted value against an independent recomputation (eval pass over the same batches, hand-computed mean). Check model.train()/eval() state at log time and confirm the logged tensor is the optimized tensor. Prediction if H3: raw batches descend (or diverge) while the reported average stalls — the curve, not the training, is broken.
2. Run the overfit probe (H2 vs. H1). Train on a single 64-sample batch with no regularization, no scheduler, no augmentation. Prediction if pipeline-healthy: loss → ~0 within tens of steps. Failure to overfit exonerates LR schedules entirely — the defect is structural (data decorrelation or gradient blockage), and no sweep will fix it. Follow with the label-shuffle control: randomize labels on the same subset; if “training” behaves identically, the model was never reading the labels (H2).
3. Sweep one optimizer variable (H1) only after H3/H2 exoneration. LR ×10 and ÷10 as two separate single-variable runs from the same checkpoint state, gradient norms logged per step. Prediction if H1-LR: the loss-curve slope changes sign/magnitude monotonically with LR (too-low flat → responsive slope; too-high oscillation/NaN with pre-NaN gradient explosion recorded).
4. Treat loss values as MEASUREMENT, curve shapes as evidence, causes as HYPOTHESIS until the intervention moves the curve as predicted. A diagnosis is incomplete until it predicts the next curve (book invariant).

    flowchart TD
    B["quantify the symptom: per-step loss + grad norm, 50 steps, no edits"] --> I["H3: log raw per-batch loss beside the reported curve; check train/eval mode + tensor identity"]
    I --> ID{"raw batches move while the reported curve stalls?"}
    ID -->|yes| H3["H3 convicted: instrumentation — fix logging, add an independent-metric assertion"]
    ID -->|no| O["overfit one 64-sample batch: plain SGD, no scheduler, no augmentation"]
    O --> OD{"loss reaches ~0 within tens of steps?"}
    OD -->|"no, grad norm ~ 0"| H1g["H1: gradient flow blocked — not an LR problem"]
    OD -->|"no, healthy norms; label-shuffle behaves identically"| H2["H2: the data carries no learnable signal"]
    OD -->|yes| S["pipeline healthy — sweep one optimizer variable: LR x10 and /10 from identical state"]
    S --> SD{"curve slope tracks LR monotonically?"}
    SD -->|yes| H1lr["H1: LR / schedule (or fp16 overflow if NaN with large but stable norms)"]
  

# overfit probe: the cheapest discriminating experiment in deep learning
subset = take(train_set, 64)          # fixed 64 samples, shuffle OFF, augment OFF
model, opt = fresh_model(), SGD(lr=1e-2)  # plain SGD, no scheduler, no wd
for step in range(200):
    loss = train_step(model, opt, subset)   # full 64 every step
    log(step, loss.item(), grad_norm(model))  # MEASUREMENT: loss + gradient norm
# Prediction healthy-pipeline: loss -> ~0 by step ~100, grad_norm nonzero throughout.
# Prediction H2 (data): loss stalls high with healthy norms -> inputs/labels decorrelated.
# Prediction H1-blocked (gradients): grad_norm ~= 0 from step 0 -> flow bug, not LR.
# Prediction H3: THIS loop descends while the real loop's curve stays flat -> logging bug.

OBSERVATION (constructed illustration, not a measured run): raw per-batch losses descended 2.3 → 1.1 while the epoch-average plot stayed 2.30 — the average divided by a hardcoded 500 instead of the true step count. Overfit probe on the side descended to 0.004 in 80 steps. UPDATED BELIEF: H3 supported — instrumentation bug; H1/H2 suspended without a single LR sweep or data rebuild. INFERENCE: the fix is a three-line logging correction plus an independent-metric assertion — zero GPU-hours owed. The flat curve was journalism, not training.

Research lineage: the triage taxonomy has been automated

The failure signatures are a fixed, monitorable set. Wardat and colleagues’ DeepDiagnosis instruments a training run and periodically checks for eight error conditions — dead, vanishing, and exploding gradients; numerical error (NaN/Inf); loss not decreasing; accuracy not improving; saturated activations; and more — then reports the symptom, localizes it to a layer or hyperparameter, and suggests an actionable fix (Wardat, Dantas Cruz, Le & Rajan, 2022). Their earlier DeepLocalize did the localization half by analyzing the values propagated between layers during training (Wardat, Le & Rajan, 2021). This chapter’s H1 signature list (grad.norm ≈ 0, pre-NaN explosion, LR collapsed to zero, non-descending loss) is essentially their monitor set, run by hand.

Expert debugging heuristics have been codified. Schoop, Huang, and Hartmann interviewed experts, catalogued the errors novices hit, and built UMLAUT, which checks a training program’s structure and behavior against those heuristics and returns plain-language messages linked to fixes and tutorials; in their study it measurably helped non-experts correct model bugs (Schoop, Huang & Hartmann, 2021). The Training Run Explorer in this chapter is a manual version of that instrument, and where UMLAUT or DeepDiagnosis is available it should run first.

Detection has been pushed all the way to repair. Zhang, Zhai, Ma, and Shen’s AUTOTRAINER monitors a training run for five problems — vanishing gradient, exploding gradient, dying ReLU, oscillating loss, slow convergence — and, on detecting one, patches the architecture or hyperparameters and continues training from the current state; on their curated set of buggy models it detected every seeded problem and repaired most of them (Zhang et al., 2021). This is the far end of the automation spectrum, and it sits in tension with this book’s stance: AUTOTRAINER disposes — it changes the model without a human confirming the diagnosis. Used as a detector it is another monitor set; used as an auto-repairer it should still produce the mechanism line and the intervention for a human to sign off before the patched run is trusted.

The NaN case has a standard fix worth naming. Exploding gradients before a NaN — the H1 signature after the LR crank — are the failure Pascanu, Mikolov, and Bengio addressed with gradient-norm clipping (Pascanu, Mikolov & Bengio, 2013); logging grad.norm per step is both the diagnostic and the trigger for the fix.

Lab 15: one-variable intervention with predicted curve change

PROPOSED, not executed: no author-measured results are reported. The evidence this chapter requires is the reader’s own curve-intervention record.

Setup. Take any misbehaving curve (or inject one: log loss.detach().mean() over the wrong dim for H3; shuffle 100% of labels for H2; set LR to 0 via an aggressive scheduler for H1). Freeze seed, data hash, and model init — controlled variables; the single intervention is the independent variable.

Task.

1. Write H1/H2/H3 with distinct drawn curve predictions before intervening — sketch the expected next-50-steps curve per hypothesis (flat / descending / NaN-with-gradient-record). No sketch, no experiment.
2. Run instrumentation interrogation first (raw-vs-reported for 50 steps), then the overfit probe (200 steps on 64 samples,shuffle/augment off), then at most one optimizer change. One intervention per run — combined “fixed logging and bumped LR” runs are inconclusive.
3. Record OBSERVATION (loss numbers + gradient norms verbatim, or small plotted table) and UPDATED BELIEF per run. Training is stochastic — repeat the decisive run with a second seed; single-seed curve convictions are UNKNOWN.
4. The convicted intervention must predict the next curve quantitatively (FORECAST: “loss < 1.5 by step 100”) and the confirmation run must test it.

Success criterion. A named conviction (H1/H2/H3 with mechanism: e.g., “H3: epoch mean divided by hardcoded 500”) plus a confirmation run meeting its pre-written FORECAST. A descended curve after multi-variable edits is explicitly not completion — attribution is the lab.

Companion tool: Training Run Explorer

What it accepts: per-step loss + gradient-norm series, the raw-vs-reported comparison, the overfit-probe series, the single-variable intervention log with pre-written FORECASTs, and seed/data/model hashes. What it performs: it plots raw against reported (H3 check), checks overfit-to-zero as a gate before any LR verdict, enforces one-intervention-per-run, and refuses a pathology verdict while any triage row is UNKNOWN or single-seeded. What it can establish: which system (instrumentation / data / optimizer) explains the curve under the examined runs — and the mechanism line (e.g., wrong divisor, shuffled labels, zeroed LR). What it cannot establish: data honesty beyond the overfit gate (Chapter 13’s audit), tensor correctness (Chapter 14), or evaluation validity (Chapter 16) — a descending curve can still descend on leakage. It never treats curve smoothness, a single seed, or downstream accuracy as diagnosis of training health. How its output changes your next action: H3 routes to logging fixes + independent-metric assertions; H2 routes to the data-path audit (Chapter 13) with the overfit table as evidence; H1 routes to optimizer/loss repair with gradient-norm records; a descending-on-leakage suspicion routes to Chapter 13 before any celebration.

Paper form, sufficient for this chapter:

Hashes (seed/data/model-init): ___ / ___ / ___
Baseline (50 steps): loss ___ -> ___   grad_norm ___ -> ___
H3 raw-vs-reported (50 steps agree? Y/N): ___   divisor/tensor audited: ___
Overfit 64-sample (200 steps): ___ -> ___   grad_norm: ___   Label-shuffle control: ___
Intervention (one): ___   FORECAST: loss ___ by step ___
Confirm (seed 2): ___   met FORECAST? Y/N
CONVICTION: H1 / H2 / H3 (circle; mechanism line: ___)

Software implementations exist: UMLAUT (Schoop et al.) and DeepDiagnosis (Wardat et al.) both check a training run against a heuristic set and suggest fixes. Where one is in the reader’s stack, run it first; where it is not, this record is the tool. The triage discipline precedes any automation.

Reusable procedure: every bad curve gets this

1. Quantify the symptom — numbers + gradient norms, 50 steps, no edits.
2. Interrogate instrumentation — raw vs. reported, tensor identity, mode flags.
3. Overfit the subset — 64 samples, plain SGD, shuffle/augment off, plus label-shuffle control.
4. Change one optimizer variable — LR ×/÷10 from identical state, norms logged.
5. Forecast and confirm — pre-written curve FORECAST, second-seed confirmation.

Failure modes

• Sweep-first triage. Launching a 20-job LR grid on an H3 logging bug. The grid “finds” an LR that changes the reported number’s noise and ships a fiction.
• Overfit skipped as trivial. “Overfitting is bad, why would I do it on purpose?” The overfit probe is a pipeline health check, not a training goal — skipping it leaves H2/H1-structural live forever.
• Multi-variable rescue. Fixing logging, shuffling, and LR between runs then declaring victory. The curve improved; the cause is UNKNOWN; the bug returns next project.
• Norm-blind tuning. Changing optimizers without logging gradient norms. Norms separate “no signal” (H2) from “no flow” (H1-blocked) from “wrong step” (H1-LR) — without them all three look “flat.”
• Smoothed-curve reading. Diagnosing from exponentially-smoothed TensorBoard lines. Smoothing hides oscillation, NaN onsets, and step artifacts — read raw points for diagnosis, smoothed lines for slides.
• Single-seed conviction. One descending run declared as proof. Training draws vary; the confirmation seed is part of the experiment, not optional rigor.

Limits, per contract: one triage record convicts one mechanism under one code/data/seed revision; it does not certify data honesty, evaluation validity, or future stability. UNKNOWN where norms are unlogged, trials are single, or interventions were combined.

References

• Andrej Karpathy. A Recipe for Training Neural Networks. 2019. https://karpathy.github.io/2019/04/25/recipe/
• Mohammad Wardat, Wei Le, and Hridesh Rajan. DeepLocalize: Fault Localization for Deep Neural Networks. Proceedings of the 43rd International Conference on Software Engineering (ICSE), 2021, pp. 251–262. https://doi.org/10.1109/ICSE43902.2021.00034
• Mohammad Wardat, Breno Dantas Cruz, Wei Le, and Hridesh Rajan. DeepDiagnosis: Automatically Diagnosing Faults and Recommending Actionable Fixes in Deep Learning Programs. Proceedings of the 44th International Conference on Software Engineering (ICSE), 2022, pp. 561–572. https://doi.org/10.1145/3510003.3510071
• Eldon Schoop, Forrest Huang, and Björn Hartmann. UMLAUT: Debugging Deep Learning Programs using Program Structure and Model Behavior. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, 2021. https://doi.org/10.1145/3411764.3445538
• Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the Difficulty of Training Recurrent Neural Networks. Proceedings of the 30th International Conference on Machine Learning (ICML), 2013, pp. 1310–1318. https://proceedings.mlr.press/v28/pascanu13.html
• Xiaoyu Zhang, Juan Zhai, Shiqing Ma, and Chao Shen. AUTOTRAINER: An Automatic DNN Training Problem Detection and Repair System. Proceedings of the 43rd International Conference on Software Engineering (ICSE), 2021, pp. 359–371. https://doi.org/10.1109/ICSE43902.2021.00043

Debugging Checklist

• Baseline symptom quantified (loss + grad norms, 50 steps, no edits)?
• Raw-vs-reported comparison completed before any training change (H3 row)?
• Overfit 64-sample probe + label-shuffle control run (H2 row)?
• At most one optimizer variable changed per run with pre-drawn curve FORECAST?
• Decisive run confirmed on a second seed?
• Conviction names the mechanism line, not just the system?
• No smoothed-only reading or multi-variable rescue accepted?

What This Chapter Established

• Training triage order H3 → H2 → H1 (instrumentation → data → optimizer) with the overfit probe as the structural gate and gradient norms as the separator. The loss-at-init and overfit-a-batch checks are canonical (Karpathy); the H1 signature set is the monitor set of DeepDiagnosis/DeepLocalize (Wardat et al.), with half-precision overflow (fixed by loss scaling / bf16, not LR) a distinct NaN cause; UMLAUT (Schoop et al.) codifies the same expert heuristics into automated checks, and AUTOTRAINER (Zhang et al.) pushes them to auto-repair — which disposes, so its output still needs a human-signed mechanism line.
• The one-variable intervention discipline with pre-drawn curve FORECASTs and second-seed confirmation, demonstrated on the flat-at-2.303 case convicted as a logging-divisor H3 — constructed illustration, no measured runs claimed.
• Lab 15 as a proposed curve-intervention record the reader executes; the Training Run Explorer contract (accepts/performs/can-establish/cannot-establish/next-action).
• What was NOT proved: data honesty beyond the overfit gate, tensor correctness, or that a descending curve descends on legitimate signal.
• Forward link: the curve now descends honestly to an excellent score — and excellence is the next suspect. A honestly-trained model with a dishonest evaluation is a passing grade on the wrong exam. That exam is next.

Next

Loss descends, gradients flow, the overfit gate passes, validation accuracy is 0.97 — and the model is broken. Not mis-trained: mis-measured. The split leaks, the metric rewards the wrong behavior, or a single lucky seed bought the headline number. Training debugging ends where evaluation debugging begins: the score says pass, the intent says fail, and only one of them is the customer.