Training Regressions and Reproducible Experiments: When Nothing Crashes but the Model Gets Worse
PyTorch: Zero to Hero — Advanced Step 09B
Most bugs in this book have announced themselves loudly.
A tensor had the wrong shape.
A parameter had no gradient.
A model refused to learn.
CUDA ran out of memory.
A compiled graph broke or recompiled.
But some of the most expensive machine-learning bugs are quieter.
The program still runs.
The tests still pass.
Training still finishes.
And yet:
validation loss is worse
training takes 12% longer
peak VRAM is 20% higher
gradients became less stable
compile count increased
throughput dropped
convergence needs twice as many steps
Nothing failed inside the run.
The run itself became worse.
That is a regression.
The central idea of this chapter is:
You cannot debug a regression unless you can make two runs meaningfully comparable.
That means controlling what you can, recording what you cannot, and treating the measurement system as code that can itself be wrong.
1. A single training run has no context
Suppose your model reports:
validation loss = 2.43
Is that good?
There is no answer yet.
Perhaps yesterday it was:
2.71
Then 2.43 looks good.
Perhaps yesterday it was:
2.18
Then something may have regressed.
A metric becomes useful when we can compare it against something meaningful.
That gives us the basic experimental pattern:
baseline
↓
make one intentional change
↓
run the same measurement
↓
compare
This sounds obvious.
In practice, people often change several things at once and then try to explain the result afterwards.
2. Change one thing when you can
Imagine changing all of these together:
batch size
learning rate
optimizer
model width
PyTorch version
mixed precision
compiler settings
Then validation loss improves.
Which change caused it?
You do not know.
Now imagine:
baseline
↓
change optimizer only
↓
measure
↓
keep or revert
↓
change learning rate only
↓
measure
This is slower than changing everything at once.
It is much faster than spending two days explaining an uninterpretable result.
The principle is:
Make the smallest change that can test the hypothesis.
3. Build a run record
A useful experiment should leave evidence behind.
Start with a small record:
from dataclasses import dataclass, asdict
from pathlib import Path
import json
import platform
import torch
@dataclass
class RunRecord:
name: str
seed: int
batch_size: int
learning_rate: float
epochs: int
torch_version: str
python_version: str
cuda_version: str | None
device: str
final_train_loss: float
final_val_loss: float
examples_per_second: float
peak_memory_mib: float | None
def save(self, path):
Path(path).write_text(
json.dumps(asdict(self), indent=2)
)
The exact schema will vary.
The important point is that the result and the conditions that produced it live together.
A number without its conditions is weak evidence.
4. Record the environment too
Machine-learning behavior depends on more than Python source code.
At minimum, useful run metadata can include:
Python version
PyTorch version
CUDA runtime
GPU model
dtype / precision
seed
batch size
sequence length
optimizer
learning rate
model configuration
dataset version
checkpoint / git commit
compiler settings
A small helper:
import platform
import torch
def environment_info():
info = {
"python": platform.python_version(),
"torch": torch.__version__,
"cuda_available": torch.cuda.is_available(),
"cuda_runtime": torch.version.cuda,
}
if torch.cuda.is_available():
info["device"] = torch.cuda.get_device_name(0)
return info
If a regression appears only after changing PyTorch or CUDA, that is very different from a regression caused by model code.
5. Randomness is part of the experiment
Training often contains randomness in:
parameter initialization
batch order
data augmentation
dropout
sampling
some accelerator kernels
If two runs use different randomness, their difference may be partly noise.
Start by setting seeds where appropriate:
import random
import numpy as np
import torch
seed = 42
random.seed(seed)
np.random.seed(seed)
torch.manual_seed(seed)
If CUDA is involved, PyTorch’s random-number state is also part of the picture.
But a seed is not a magic reproducibility switch.
6. Reproducibility is not the same as determinism
These ideas are related but different.
Reproducibility asks:
Can I recreate a sufficiently comparable result?
Determinism asks:
Given the same inputs and conditions, do operations return the same result every time?
PyTorch can request deterministic algorithms:
torch.use_deterministic_algorithms(True)
This can be extremely useful when chasing a numerical difference.
But deterministic execution can have costs.
Some operations may become slower.
Some nondeterministic operations may raise an error when a deterministic implementation is unavailable.
And deterministic settings do not guarantee identical results across every PyTorch release, device or platform.
So use determinism as a debugging tool, not as a ritual.
7. A practical reproducibility setup
For an experiment where repeatability matters, you might begin with:
import random
import numpy as np
import torch
def set_reproducible_seed(seed: int):
random.seed(seed)
np.random.seed(seed)
torch.manual_seed(seed)
if torch.cuda.is_available():
torch.cuda.manual_seed_all(seed)
Then, when you need stronger deterministic behavior:
torch.use_deterministic_algorithms(True)
For cuDNN workloads, benchmark selection can also introduce variability, so during strict debugging you may choose settings that prioritize repeatability over maximum autotuned speed.
The rule is not:
always force every deterministic setting
It is:
know which sources of variation matter for the question you are asking
8. Do not confuse natural variance with regression
Suppose five baseline runs produce validation losses:
2.31
2.29
2.34
2.30
2.33
Then a new result of:
2.32
is probably not interesting.
A new result of:
2.61
is much more suspicious.
One run is often not enough to establish a baseline for a noisy training process.
Repeat important experiments.
Then record simple statistics.
import statistics
losses = [2.31, 2.29, 2.34, 2.30, 2.33]
print("mean:", statistics.mean(losses))
print("stdev:", statistics.stdev(losses))
You do not need a statistics degree to stop treating normal variation as a bug.
9. Use distributions, not just one number
Performance is noisy too.
Instead of:
step took 12.4 ms
measure many steps:
import statistics
samples_ms = [
benchmark_one_step()
for _ in range(50)
]
print({
"median_ms": statistics.median(samples_ms),
"mean_ms": statistics.mean(samples_ms),
"min_ms": min(samples_ms),
"max_ms": max(samples_ms),
})
For production latency you may also care about tail values such as p95 or p99.
For local training throughput, median or a trimmed average is often already much better than one timing sample.
10. Build a baseline before optimizing
Suppose we want to test mixed precision.
Bad experiment:
turn on autocast
change batch size
change optimizer
change DataLoader workers
compile the model
compare against memory of yesterday's run
Better experiment:
BASELINE
fp32
batch 64
same model
same dataset
same seed policy
same benchmark
EXPERIMENT
AMP
batch 64
same model
same dataset
same seed policy
same benchmark
Then compare:
validation loss
steps/sec
peak memory
That gives you a result you can reason about.
11. Measure several dimensions at once
A change can improve one metric while hurting another.
For training, a useful scorecard might include:
train loss
validation loss
accuracy or task metric
steps/sec
tokens/sec
peak VRAM
compile time
number of recompiles
NaN / Inf count
gradient norm
For example:
baseline candidate
----------------------------------------
val loss 2.31 2.30
steps/sec 8.4 9.7
peak VRAM 11.2 GiB 14.9 GiB
Is the candidate better?
That depends on your constraint.
If memory limits batch size, the answer may be no.
This is why “faster” or “better” should usually name a metric.
12. Separate capability tests from regression tests
There are two different questions we can ask about a model.
A capability test asks:
Can the model do something it could not reliably do before?
A regression test asks:
Can the model still do something we already expect it to do?
For our small language model, capability tests might be:
Can validation loss fall below 2.5?
Can a larger context improve the task?
Can the model learn a longer dependency?
Regression tests might be:
Does training loss still decrease?
Are gradients finite?
Does generation still produce the requested number of tokens?
Did throughput fall by more than 15%?
Did peak memory increase by more than 20%?
These suites have different personalities.
Capability tests should contain room to improve.
Regression tests should protect behavior we already trust.
13. Successful capabilities can graduate into regressions
Suppose this was once difficult:
validation loss < 2.5
After several improvements, every healthy run reaches:
2.2 ± 0.03
Then the old capability can become a regression expectation.
The question changes from:
Can we reach 2.5?
to:
Why did this build suddenly fail to reach 2.5?
This is how an experiment becomes a guardrail.
Failures discovered during development should gradually become tests.
14. Regression thresholds should reflect noise
This is fragile:
assert current_ms <= 4.200
especially on shared hardware.
A threshold should reflect the stability of the measurement environment.
For example:
allowed_regression = 0.15
regression = current_ms / baseline_ms - 1
assert regression <= allowed_regression
Even that may be inappropriate on a noisy shared CI runner.
For unstable hardware, you might:
record the result without failing CI
run dedicated benchmark hardware
compare distributions
require repeated evidence before alerting
A regression gate that constantly cries wolf will eventually be ignored.
15. Debug the benchmark too
This may be the most important section in the chapter.
Suppose version B appears worse:
A validation loss = 2.31
B validation loss = 2.54
Before declaring a model regression, ask:
same validation examples?
same preprocessing?
same tokenizer?
same masking?
same sequence lengths?
same metric aggregation?
same model.eval() state?
same checkpoint loading?
same data leakage rules?
The benchmark is software.
Therefore the benchmark can contain bugs.
A measurement failure can look exactly like a model failure.
16. A broken evaluation can produce very convincing numbers
Imagine accidentally changing the validation loader from:
shuffle=False
to a pipeline that applies training augmentation.
Or imagine changing tokenization so padding is now included in the loss.
The final metric may move dramatically.
The model did not necessarily regress.
The meaning of the metric changed.
That gives us another debugging split:
MODEL CHANGED
→ behavior may have changed
MEASUREMENT CHANGED
→ reported behavior may have changed
When a result surprises you, investigate both.
17. Freeze an evaluation set
For regression work, keep a stable evaluation sample.
For example:
from torch.utils.data import Subset
fixed_indices = [
3, 17, 22, 41, 58,
73, 91, 104, 138, 155,
]
regression_set = Subset(
validation_dataset,
fixed_indices,
)
For a real project you would normally use a larger and more representative set.
The point is that the comparison set should not silently change between commits.
Version it if necessary.
18. Inspect individual failures
An aggregate score tells you that something changed.
It rarely tells you why.
Suppose validation accuracy falls from:
91.2% → 88.7%
Do not stop there.
Look at examples that changed from correct to incorrect.
failures = []
for x, y in validation_loader:
logits = model(x)
pred = logits.argmax(dim=-1)
mask = pred != y
for item_x, item_y, item_pred in zip(
x[mask], y[mask], pred[mask]
):
failures.append((item_x, item_y, item_pred))
Then ask:
same class failing repeatedly?
long sequences only?
small images only?
rare labels?
all failures after one preprocessing change?
Aggregate metrics detect.
Individual examples explain.
19. Compare curves, not just final loss
Two runs can finish at similar loss while behaving very differently.
Run A:
fast early learning
stable convergence
Run B:
slow learning
large oscillations
late recovery
If you only compare the final value, you may miss a real regression in optimization stability.
Keep the curve.
history = {
"train_loss": [],
"val_loss": [],
}
Then compare by step or epoch.
Useful questions include:
When did the runs diverge?
Did one become unstable immediately?
Did a regression begin after the learning-rate change?
Did validation worsen while training improved?
Time is part of the evidence.
20. Gradient statistics can reveal hidden regressions
A model may still reduce loss while gradients become unhealthy.
Track simple statistics:
import math
def grad_l2_norm(model):
total = 0.0
for p in model.parameters():
if p.grad is None:
continue
value = p.grad.detach().float().norm(2).item()
total += value * value
return math.sqrt(total)
Then record:
step
loss
gradient norm
A code change that suddenly causes gradient norms to explode may be worth investigating before the final metric collapses.
Regression monitoring can catch precursors, not only final failures.
21. Performance regressions deserve first-class treatment
We already built a performance harness in the previous chapter.
Now keep historical baselines.
baseline = {
"steps_per_second": 8.4,
"peak_memory_mib": 11264,
}
current = {
"steps_per_second": 7.1,
"peak_memory_mib": 13780,
}
Calculate deltas:
throughput_change = (
current["steps_per_second"]
/ baseline["steps_per_second"]
- 1
)
memory_change = (
current["peak_memory_mib"]
/ baseline["peak_memory_mib"]
- 1
)
print({
"throughput_change": throughput_change,
"memory_change": memory_change,
})
A program can remain functionally correct while becoming economically unusable.
Performance is behavior too.
22. Compiler regressions are another dimension
After the previous chapter, we can add compiler health to the run record.
For example:
first-call compile latency
steady-state latency
number of graph breaks
number of recompiles
Imagine:
baseline recompiles = 1
candidate recompiles = 37
The model may still produce identical outputs.
But a change in shape behavior or Python control flow has altered the execution system dramatically.
That is a regression worth catching.
23. Keep code and data versions together
A training result depends on both.
Useful identifiers include:
git commit
dataset hash or version
tokenizer version
configuration hash
checkpoint ID
For example:
import hashlib
import json
def config_hash(config: dict) -> str:
payload = json.dumps(
config,
sort_keys=True,
).encode()
return hashlib.sha256(payload).hexdigest()[:12]
Now a result can say:
commit=8ac31d2
config=1b9f420a9c31
dataset=v4
That is much better than:
I think I used the same settings.
24. Build an experiment result object
Put the ideas together:
from dataclasses import dataclass, asdict
import json
@dataclass
class ExperimentResult:
name: str
seed: int
config_hash: str
train_loss: float
val_loss: float
steps_per_second: float
peak_memory_mib: float | None
gradient_norm: float | None
def to_json(self):
return json.dumps(
asdict(self),
indent=2,
)
Then save every meaningful experiment.
Now comparison becomes data rather than recollection.
25. Compare experiments automatically
def compare(baseline, candidate):
return {
"val_loss_delta": (
candidate.val_loss
- baseline.val_loss
),
"throughput_delta_pct": (
candidate.steps_per_second
/ baseline.steps_per_second
- 1
) * 100,
"memory_delta_pct": (
candidate.peak_memory_mib
/ baseline.peak_memory_mib
- 1
) * 100,
}
Now a report can say:
validation loss: +0.04
throughput: -11.8%
peak memory: +18.2%
That is much easier to reason about than two pages of training logs.
26. Define what counts as meaningful before you run
If you decide the success criterion after seeing the result, it is easy to rationalize anything.
Before running an experiment, write something like:
Hypothesis:
AMP will improve throughput without materially worsening validation loss.
Success criteria:
throughput improves >= 10%
validation loss worsens <= 0.02
peak memory does not increase
Then run it.
This turns experimentation into a testable claim.
It also makes failed experiments useful.
27. A failed experiment can still be a good result
Suppose your hypothesis is:
batch size 128 will improve throughput
The experiment reports:
throughput +4%
peak memory +70%
validation unchanged
That may fail your success criterion.
But you learned something real.
The failure becomes evidence about the system.
This is much better than silently discarding runs that did not confirm your intuition.
28. Keep a tiny experiment ledger
Even a Markdown table helps:
| ID | Change | Val loss | Steps/s | VRAM | Result |
|----|--------|----------|---------|------|--------|
| A0 | baseline | 2.31 | 8.4 | 11.2G | baseline |
| A1 | AMP | 2.32 | 9.8 | 8.1G | keep |
| A2 | compile | 2.32 | 10.4 | 8.4G | keep |
| A3 | batch 128 | 2.35 | 10.6 | 15.9G | reject |
This may look primitive.
It is still vastly better than relying on memory.
29. Regression debugging is causal debugging
Suppose a regression begins between commit A and commit B.
The goal is to narrow the causal difference.
If the commits are large, bisect.
If the configuration changed, restore one value at a time.
If the dataset changed, rerun the old code on the new data and the new code on the old data if possible.
Conceptually:
old code + old data → baseline
new code + old data → isolates code change
old code + new data → isolates data change
new code + new data → current system
Not every project can reproduce all four cells.
But thinking in those terms prevents vague explanations.
30. Watch for train/eval mode mistakes
A surprising regression sometimes comes from evaluation code rather than model code.
Always make evaluation mode explicit:
model.eval()
with torch.inference_mode():
for x, y in validation_loader:
logits = model(x)
If dropout or batch normalization remains in training mode, the evaluation distribution changes.
This is another example of benchmark validity.
Before blaming the model, verify the measurement path.
31. Watch for data-order changes
Changing:
shuffle=True
or DataLoader worker behavior can alter training randomness even when the dataset itself is unchanged.
That may be fine.
But it means:
run A != run B
in more ways than the code diff suggests.
For exact debugging, stabilize the input order.
For robustness testing, deliberately vary it and examine the distribution of results.
Those are different experiments.
32. Watch for preprocessing drift
Model code is often innocent.
Examples of preprocessing regressions:
normalization constant changed
image resize changed
label mapping changed
tokenizer changed
padding changed
masking changed
augmentation accidentally applied to validation
Record preprocessing configuration with the model configuration.
The data pipeline is part of the model system whether or not it lives in nn.Module.
33. A regression suite should be small enough to run
It is tempting to make every training test enormous.
Then nobody runs it.
Use layers.
FAST REGRESSION
seconds / a few minutes
basic loss movement
finite gradients
shape contracts
small fixed evaluation sample
MEDIUM REGRESSION
short training run
throughput
peak memory
validation metric
FULL EXPERIMENT
complete training schedule
large validation suite
statistical comparison
The fastest useful signal should arrive first.
34. A tiny smoke-training regression test
A model should at least be able to overfit a tiny batch.
def tiny_overfit_test(
model,
x,
y,
steps=100,
lr=1e-2,
):
optimizer = torch.optim.AdamW(
model.parameters(),
lr=lr,
)
losses = []
for _ in range(steps):
optimizer.zero_grad(set_to_none=True)
logits = model(x)
loss = torch.nn.functional.cross_entropy(
logits,
y,
)
loss.backward()
optimizer.step()
losses.append(loss.item())
return losses
Then assert something coarse:
losses = tiny_overfit_test(model, x, y)
assert losses[-1] < losses[0] * 0.5
This is not a scientific benchmark.
It is a regression smoke test for the training mechanism.
35. Test invariants as well as metrics
Some properties should simply remain true.
Examples:
assert torch.isfinite(loss)
for p in model.parameters():
if p.grad is not None:
assert torch.isfinite(p.grad).all()
assert logits.shape == (batch_size, num_classes)
assert peak_memory_mib < memory_budget_mib
These do not replace quality metrics.
They catch different classes of regression.
36. Use tolerances, not fake precision
Floating-point systems do not owe you exact bitwise equality unless you have engineered for it.
For output comparisons use tolerances:
torch.testing.assert_close(
actual,
expected,
rtol=1e-4,
atol=1e-5,
)
For metrics use ranges appropriate to natural variance.
The purpose of a regression test is to detect meaningful change.
Not to punish harmless floating-point noise.
37. Re-run surprising results
A single strange run may be:
real regression
random variation
thermal throttling
background load
bad batch
transient I/O problem
measurement bug
So when the result is surprising:
repeat it
If it disappears, investigate variance.
If it repeats, confidence increases.
If it appears only on one environment, that is evidence too.
38. Keep the baseline honest
A baseline can become stale.
Suppose hardware, PyTorch, data and model architecture all evolve.
Comparing forever against a six-month-old baseline may stop being useful.
When adopting a new known-good configuration, promote it deliberately:
old baseline
↓
validated candidate
↓
new baseline
Record when and why the baseline changed.
Otherwise “regression” gradually loses meaning.
39. Build a regression report
A compact report can look like:
RUN: candidate-184
BASELINE: main-172
CORRECTNESS
validation loss 2.31 → 2.34 +0.03
accuracy 91.2% → 90.8% -0.4pp
finite gradients yes → yes
PERFORMANCE
steps/sec 8.4 → 7.6 -9.5%
peak VRAM 11.2G → 12.8G +14.3%
COMPILER
recompiles 1 → 14
ENVIRONMENT
torch same
CUDA same
GPU same
seed policy same
Now the debugging conversation starts from evidence.
40. A practical regression-debugging ladder
When a model gets worse without crashing, work through this sequence:
flowchart TD
A[Confirm the metric changed] --> B[Repeat the run]
B --> C[Verify benchmark/eval path]
C --> D[Compare config + environment]
D --> E[Check seed / data order / preprocessing]
E --> F[Inspect individual failures]
F --> G[Compare learning curves]
G --> H[Compare gradient + numerical stats]
H --> I[Compare throughput + memory]
I --> J[Compare compiler behavior]
J --> K[Isolate smallest causal change]
K --> L[Add regression test]
In text:
1. confirm the regression
2. repeat it
3. validate the benchmark itself
4. compare environment and configuration
5. control randomness where useful
6. inspect individual failing examples
7. compare curves, not only final values
8. inspect numerical signals such as gradient norms
9. inspect performance and memory
10. inspect compile/recompile behavior
11. isolate the smallest causal difference
12. turn the discovered failure into a regression test
That is the core workflow.
41. Challenge: create five regressions deliberately
Take a small training project and introduce these changes one at a time.
Regression 1: learning rate
Multiply the learning rate by ten.
Record:
loss curve
gradient norm
final validation loss
Regression 2: DataLoader
Set:
num_workers = 0
Record throughput.
Regression 3: memory
Double sequence length or image resolution.
Record peak memory.
Regression 4: evaluation bug
Accidentally leave the model in training mode during validation.
Observe how the metric changes.
Then fix the benchmark rather than the model.
Regression 5: compilation
Introduce shape variation that causes repeated compilation.
Record first-call latency and recompilation logs.
The point is to experience five different meanings of:
the new version is worse.
42. Challenge: build a reproducible experiment folder
For one experiment, save:
config.json
environment.json
metrics.json
loss_curve.csv
notes.md
Your notes.md should contain:
hypothesis
single intentional change
success criteria
result
interpretation
next experiment
Now imagine returning to the experiment six months later.
Could you explain what happened without relying on memory?
If yes, your experiment is becoming durable engineering work.
43. The deeper lesson
Machine learning encourages us to stare at final scores.
But the important object is not one number.
It is the causal chain that produced the number:
code
+
configuration
+
data
+
randomness
+
environment
+
measurement
↓
result
A regression means something in that chain changed enough to alter the result.
The debugging job is to identify which part.
That is why reproducibility matters.
Not because every run must be perfectly identical.
Because comparisons need enough shared structure to support an explanation.
44. The programmer owns the measurement boundary
By this point in the book, PyTorch can do a remarkable amount for us.
It can:
build tensor programs
calculate gradients
register model structure
feed accelerators
compile graphs
generate optimized kernels
But it cannot decide whether your experiment answered the question you intended to ask.
That remains your job.
You decide:
what the baseline is
what changed
what counts as success
what should remain invariant
which variation is acceptable
whether the benchmark is valid
whether the result is reproducible enough to trust
That is not administrative work around machine learning.
It is part of machine learning engineering.
Where the series goes next
Now we have the machinery to build the final model without treating success as:
script finished without an exception
We can ask much better questions:
Does it learn?
Can we explain its tensors?
Are its gradients healthy?
How much memory does it use?
How fast does it run?
Does compilation help?
Can we reproduce the result?
Did the latest change make it better or worse?
With those questions available, we are ready for the capstone:
Build a Small GPT-Style Language Model From Scratch
The goal is no longer merely to make a transformer run.
It is to build one we know how to investigate.
