This appendix introduces no new PyTorch mechanism.

Everything here was earned earlier in the book by building something small, breaking it deliberately, measuring what changed, and locating the first place where reality stopped matching the intended computation.

The purpose of this appendix is different.

When a real model is failing, you usually do not need another explanation of autograd, broadcasting, attention or CUDA. You need to answer a narrower question:

Which instrument should I reach for first?

The book’s recurring move was:

predict
    ↓
observe
    ↓
find the first divergence
    ↓
identify the mechanism
    ↓
change one thing
    ↓
verify the repair

That is the default procedure. The sections below route common symptoms to the smallest instrument that can produce useful evidence.

The central rule remains:

Make hidden structure visible before guessing.

Start here: route the symptom

Do not interpret this table as a strict sequence that every bug must follow.

It is a routing table.

A single failure can cross several systems. A transform bug may first appear as a training failure. An optimizer-ownership bug may first appear as a flat loss. A compiler problem may first appear as a performance regression.

The job is to move from the visible symptom to the earliest unsupported claim.

The minimum evidence packet

Before changing code, capture enough state that the failure can survive your first attempted fix.

For a training problem, the minimum useful packet is often:

one known batch
input shape / dtype / device / range
target shape / dtype / meaning
model mode
loss value
loss.requires_grad
loss.grad_fn
expected trainable parameter names
grad state for those parameters
optimizer parameter membership
parameter deltas after one controlled step
current learning rate for every parameter group

For a performance problem:

workload
input shapes
dtype
device
train/eval/inference mode
eager/compiled
metric
warmup
measurement window
repetitions
synchronization boundary
peak allocated memory
peak reserved memory

For a regression:

code version
configuration
data version
seed
environment
evaluation contract
baseline variation
paired deltas if pairing is valid

The exact packet changes with the question.

The principle does not:

Preserve observations before your intervention destroys the failing state.

1. The training loop: separate forward, backward and update

When the system is still small enough, begin with the decomposition from Chapter 1:

forward     compute prediction and loss
backward    compute derivatives and accumulate them into .grad
update      change parameter values

These are separate mechanisms.

loss.backward() does not update parameters.

An update does not clear gradients.

A falling loss does not prove that the intended task is being learned.

Ask

What value produced this loss?
What gradient was produced by this loss?
What parameter value existed before the update?
What parameter value exists after the update?
Was the old gradient cleared before the next backward pass?
Is this still the same parameter object?

High-value measurements

before = p.detach().clone()

loss.backward()

grad = None if p.grad is None else p.grad.detach().clone()

optimizer.step()

delta = p.detach() - before

For a controlled plain-SGD step with no momentum or weight decay:

delta ≈ -learning_rate * gradient

If that equality fails under the deliberately restricted experiment, the update mechanism is not the one you think it is.

If it holds, do not generalize it blindly to Adam, momentum, weight decay or other stateful update rules.

Stop when

You can separately prove:

the forward calculation is the intended one
backward produced the expected derivative
the intended parameter object moved
old gradients are not contaminating the next step

Then move to the next layer of the system.

2. Tensors: find the first wrong tensor, not the first illegal one

A tensor is not only values.

The useful description is:

shape
dtype
device
stride
contiguity
axis meaning
value meaning

The first five are inspectable from the tensor.

The last two belong to the program’s contract.

First move

Write the expected tensor before running the operation.

Not:

I expect something like [32, ?]

but:

[B, T, D]
B = examples
T = positions
D = feature coordinates

Then compare expectation with observation at each boundary.

Use deterministic values when layout is the problem

Random numbers hide rearrangements.

For reshape, transpose, head splitting or permutation bugs, use:

x = torch.arange(...)

small enough to print.

If two operations produce the same shape, inspect where the values moved.

Common false repairs

reshape until the exception disappears
change in_features to match the tensor that arrived
change in_channels to match an NHWC batch
squeeze dimensions without naming what they represent

These can turn an upstream mistake into a legal downstream architecture.

Stop when

You have found the first boundary where observed tensor facts or axis semantics stop matching the expected contract.

Repair there.

3. Autograd: find where the dependency path stops existing

When a parameter does not receive the expected gradient, do not begin with the final .grad.

Start from the question:

Does this loss actually depend on this parameter in the graph that executed?

Autograd records the operations that ran during this forward pass. It does not record the model you intended to run.

Evidence ladder

parameter.requires_grad
parameter.is_leaf
loss.requires_grad
loss.grad_fn
parameter.grad after a controlled backward pass
reachability from loss back to expected leaves

A healthy loss.grad_fn proves that the loss has some differentiable history.

It does not prove that every intended parameter lies on that history.

Distinguish

grad is None

from:

grad is a tensor of zeros

Under a controlled pass that begins with .grad = None:

• None means no gradient was accumulated into that parameter;
• a zero tensor means autograd materialized a gradient whose accumulated derivative is zero.

Those are different mechanisms.

Common causes of a missing path

detach()
torch.no_grad() around part of the forward computation
a branch that did not execute
a parameter that was not used
rebinding to a new non-tracked tensor

Stop when

You can point to the first operation or branch after which the intended dependency no longer exists.

4. A raw neural network: ask which of six claims is failing

Chapter 4 established that “the gradients look fine” answers only one question.

For a network that is mechanically running, classify the failure across six claims:

Why this matters

These failures can look similar:

missing nonlinearity
symmetric initialization
zero initialization
detached parameter
healthy gradient on an un-updated parameter
wrong objective

No single gradient statistic distinguishes all of them.

Stop when

You can say which claim the evidence supports and which claim is still unsupported.

Do not use evidence for one claim as proof of another.

5. nn.Module: compare the four structures

When an object seems to disappear, stop thinking of “the model” as one thing.

There are four overlapping structures:

1. Python object graph
2. registered module tree
3. autograd graph
4. optimizer parameter groups

They answer different questions.

The diagnostic question

Which structure does the failing operation traverse, and is my object actually in it?

A tensor can:

require gradients
receive gradients
participate in forward

and still not be:

registered
saved
moved by model.to(...)
owned by the optimizer

Object identity matters

If a head is replaced after optimizer construction, the optimizer may still own the old parameter objects.

Names and shapes can look correct.

The identities are wrong.

Stop when

You can reconcile:

what Python can reach
what PyTorch registered
what autograd reached
what the optimizer owns

for every object relevant to the failure.

6. DataLoader and input delivery: split producer from consumer

“Low GPU utilization” is not a DataLoader diagnosis.

First ask whether the training loop is waiting for batches.

The smallest useful measurement is:

t0 = perf_counter()
batch = next(iterator)
t1 = perf_counter()

train_step(batch)
t2 = perf_counter()

Now you have two different observations:

wait = t1 - t0
work = t2 - t1

Model the pipeline

producers
    ↓
queue / buffering
    ↓
consumer

Then separate:

producer throughput
consumer throughput
buffering
end-to-end throughput

More workers can overlap waiting.

They cannot create more of a CPU resource that is already saturated.

Record tails, not only averages

For batch wait, capture:

median
p95
maximum

A low mean with a large tail can still starve the consumer repeatedly.

Sweep one control at a time

For example:

num_workers = 0, 1, 2, 4, 8

recording:

examples/s
batch-wait distribution
post-batch work time

If wait falls while consumer work rises and end-to-end throughput is unchanged, the workers are competing with the consumer for the same resource.

Stop when

You can name the actual limiting resource or phase:

read
decode
transform
collate
process startup
CPU contention
host-to-device transfer
consumer compute

Then repair that mechanism.

7. Transforms and preprocessing: trace one known sample

A preprocessing pipeline can produce a legal tensor whose meaning is wrong.

Chapter 7 separated the input contract into five categories:

1. STRUCTURE
2. NUMERIC INTERPRETATION
3. SEMANTICS
4. RELATIONSHIPS
5. PROVENANCE

They need different instruments.

The method

Find the first transform boundary where the sample stops satisfying the intended input contract.

Start with:

one sample
one process
no batching
controlled randomness

Then walk stage by stage.

Important limitation

An all-green metadata report is not a clean bill of health.

A tensor cannot tell you:

its RGB channels were interpreted as BGR
the augmentation invalidated the label
the mask no longer aligns with the image
normalization statistics were fitted on validation data
the serving tokenizer differs from training

Those need explicit probes or records.

Stop when

You have identified the earliest stage where one of the five contract categories fails.

Only then reintroduce batching, workers and performance complexity.

8. CNN geometry: derive before execution

For convolutional networks, keep two questions separate:

CHANNEL QUESTION
C_in → C_out

SPATIAL QUESTION
H_in,W_in → H_out,W_out

C_out is declared.

Spatial size is derived.

The method

Derive the expected shape before the layer runs. Compare it with the observed shape. Investigate the first disagreement.

Use a ledger:

stage
operation
observed input
expected output
observed output
first divergence

Never repair the final Linear first

For:

mat1 and mat2 shapes cannot be multiplied

derive backward:

Where did the flattened feature count come from?
Which earlier layer changed C, H or W?
Was that change intended?

Changing in_features is correct only if the upstream geometry change was intentional.

Remember the semantic limit

A shape ledger proves composition.

It does not prove that height and width were not swapped, that preprocessing preserved meaning, or that the architecture is suitable for the task.

Stop when

Derived and observed geometry agree at every shape-changing operation up to the failing boundary.

9. Feature space: name the represented object before interpreting numbers

When a tensor is shaped:

[B, T, D]

ask:

What does one D-vector represent?
How many such vectors are present?
Which axis contains its coordinates?

nn.Linear(D_in, D_out) transforms the last axis and preserves the leading axes.

So [B,T,D] → [B,T,1] means one score per (example, position), not one score per example.

Before interpreting a scalar, name it

dot product
cosine similarity
Euclidean distance
classifier score
signed distance to a hyperplane

They are not interchangeable.

Check invariances

Ask what should remain unchanged when:

one vector is rescaled
w and b are scaled together
one feature coordinate changes units

If the quantity changes under something that should not matter to the interpretation, you are probably naming the quantity incorrectly.

Inspector

For a representation, useful rows include:

vector norms
per-coordinate scales
non-finite values
pairwise cosine
collapse toward zero
collapse toward one direction
dominant coordinates

Stop when

You can state:

what one vector represents
which axis holds its coordinates
what operation acts on those coordinates
what numerical quantity came out
which invariances that quantity has

10. Attention: four levels of correctness

Attention needs more than a shape check.

Use four levels:

LEVEL 1  SHAPE
         Does the tensor have the dimensions I derived?

LEVEL 2  AXIS SEMANTICS
         Do those dimensions contain the objects I intended?

LEVEL 3  NUMERICAL INVARIANTS
         Do valid query rows normalize over keys?
         Are forbidden weights zero?
         Are values finite?

LEVEL 4  BEHAVIOR
         Does a future-token intervention leave the prefix unchanged?
         Does the implementation agree with a trusted reference?

The method

derive the axes
    ↓
execute one stage
    ↓
verify the invariant introduced by that stage
    ↓
stop at the first failure

High-value checks

merge(split(x)) == x
scores shape == [B, Nh, Tq, Tk]
softmax normalizes over Tk
forbidden weights == 0
no fully blocked query unless explicitly supported
merged layout restores [B, Tq, E]
future-token intervention preserves the causal prefix within tolerance

Why Level 4 exists

Missing 1/sqrt(Dh) scaling can preserve:

shape
axis semantics
finite values
row sums

and still disagree materially with the intended attention computation.

Stop when

You know the earliest correctness level and stage that fails.

11. Training that runs but does not learn: walk the learning chain

The learning chain is:

TASK
  ↓
OBJECTIVE
  ↓
DEPENDENCY
  ↓
GRADIENT
  ↓
OPTIMIZER OWNERSHIP
  ↓
PARAMETER UPDATE
  ↓
CAPABILITY

This is the right tool when:

forward runs
loss is finite
backward runs
gradients exist
optimizer.step() runs

and the model is still useless

What each link asks

TASK

Does the target still describe the input?

OBJECTIVE

Are the outputs in the representation the loss expects?
Can one example reproduce the loss manually?

DEPENDENCY

Does the loss depend on the intended parameters?

GRADIENT

Are expected gradients present?
None, zero, finite, non-finite?

OPTIMIZER OWNERSHIP

Does the optimizer own those exact parameter objects?
Does it own stale objects that are no longer in the model?

PARAMETER UPDATE

Did the intended objects actually move?

CAPABILITY

Can the controlled system overfit a tiny fixed batch?

Important warning

The learning ledger is a state-mutating probe if it calls backward() and optimizer.step().

Run it on a disposable model/checkpoint or deliberately accept the sacrificial step.

Stop when

You find the first expected consequence that fails to appear.

Debug there.

Do not tune learning rate, architecture, precision or regularization while an earlier link remains unsupported.

12. Performance: define the comparison before optimizing

Performance is not a property of one line of code.

It is a result under a contract:

workload
hardware
software/runtime
execution mode
metric
measurement boundary

First choose the question

latency
throughput
peak memory
cold-start cost
steady-state cost

These can move in opposite directions.

The method

Measure → localize → intervene → remeasure.

Performance ledger

Capture:

WORKLOAD
  model / operation
  input shape
  dtype
  device
  mode
  eager / compiled

MEASUREMENT
  warmup
  repeats
  synchronization
  metric

TIME
  input wait
  transfer
  forward
  loss
  backward
  optimizer
  logging / synchronization

MEMORY
  allocated
  reserved
  peak allocated
  peak reserved

CUDA timing

If the device is asynchronous, host wall-clock timing without a completion boundary can measure submission rather than completed device work.

Know whether you are measuring:

host-visible latency
device elapsed time
end-to-end workload time

Memory

Keep separate:

allocated     live PyTorch-managed allocations
reserved      allocator-managed pool
peak          transient high-water mark

If memory_allocated() grows across otherwise-identical iterations, the set of live PyTorch allocations is growing. Trace ownership.

Do not call high reserved memory alone a live-tensor leak.

Profiler rule

The profiler is for localization.

The unprofiled benchmark is for the final performance comparison.

Stop when

You can state:

the metric that matters
the phase/resource that dominates it
the mechanism hypothesis
the one intervention that tests that hypothesis
the correctness property that must remain unchanged
the uninstrumented before/after result

13. torch.compile: inspect the compiler’s hidden state

A compiled call that returns the right number has proved correctness.

It has not proved:

the hot region was captured
the graph was reused
guards remained valid
recompilation stopped
steady-state performance improved
cold cost amortized

A useful progression is:

RUNS
  ↓
CORRECT
  ↓
CAPTURED
  ↓
STABLE
  ↓
WORTH IT

Begin with eager

If eager is already wrong, compilation is not the first problem.

Then compare correctness

Use a stated tolerance appropriate to the workload.

Then inspect capture

Useful evidence can include:

graph-break logs
fullgraph=True as a capture test for the exact call
structured compiler summaries

No one instrument is automatically ground truth in every version. Triangulate when tools disagree.

Then inspect stability

Capture recompilation evidence.

Ask:

Which guard failed?
shape?
dtype?
device?
Python value?
Did the workload legitimately vary on that dimension?
Did a cached specialization get reused?
Did a new specialization get created?
Did the cache limit force an unmatched call to eager?

Then ask whether it was worth it

Separate:

cold first call
warm-cache first call
steady-state call
recurring recompilation

Compute amortization when startup cost matters.

Stop when

You can name the compiler assumption that stopped holding, or establish that compilation is stable and the remaining problem is ordinary workload economics.

14. Regressions: establish the difference before explaining it

A new metric is worse.

That is an observation.

It is not yet a regression mechanism.

The first three possibilities

NOTHING MEANINGFUL CHANGED
ordinary run-to-run variation

MEASUREMENT CHANGED
different evaluation path / split / preprocessing / mode

SYSTEM CHANGED
code, configuration, data, environment, compiler state

Rule out the first two before explaining the third.

Baseline variation

Run the unchanged configuration across several seeds.

Report a distribution, not only one number.

The observed min/max spread is context.

It is not a universal statistical detection threshold.

Pair when pairing is valid

Shared seeds can make a subtle comparison more sensitive when the two runs preserve enough shared randomness for the seed effects to cancel.

Inspect:

delta(seed_i) = B(seed_i) - A(seed_i)

rather than only comparing two independent ranges.

Verify the evaluation contract

Record enough to answer:

same validation examples?
same preprocessing?
same aggregation?
same model.eval() state?
same checkpoint-selection rule?

A fingerprint proves only the fields included in the fingerprint.

Record the run

At minimum:

code version
config
data identity
seed
environment
evaluation contract
compiler state if relevant
result

Bisect carefully

Bisection assumes the good/bad predicate behaves suitably.

Changes can interact.

After a boundary implicates one change, apply that candidate alone and confirm the effect.

Stop when

Repeated evidence establishes a difference worth explaining and the measurement contract is known to be comparable.

Only then investigate the mechanism.

15. The complete model: route, do not invent a new instrument

The capstone’s lesson is that no single diagnostic certifies a large model.

The complete system contains all the earlier systems at once:

text
  ↓
tokenization
  ↓
batch/target alignment
  ↓
embeddings
  ↓
attention
  ↓
residual blocks
  ↓
logits
  ↓
objective
  ↓
autograd
  ↓
optimizer
  ↓
performance / compilation
  ↓
generation
  ↓
evaluation and comparison

When the full model fails, route the symptom.

Examples:

Generated text is nonsense but training loss is excellent
    → TASK / target alignment first

A future token changes an earlier logit
    → attention Level 4 intervention

Initial loss is wildly inconsistent with the declared initialization
    → OBJECTIVE / initialization contract

Gradients are finite but the head never moves
    → optimizer ownership + parameter delta

Training is correct but throughput is low
    → performance ledger

Compiled model pauses on new sequence lengths
    → guards / recompilation

New configuration improves one run by 0.001
    → baseline variation + paired comparison

A large model does not make the earlier instruments obsolete.

It makes choosing the correct one more important.

Symptom-to-evidence field table

This table is intentionally redundant with the chapter-by-chapter guide. Use it from the other direction: start from what you observe.

The one-bug rule

When you have several failures at once, repair one boundary and rerun.

Do not fix:

input layout
layer dimensions
learning rate
optimizer
normalization

in one edit.

A multi-change repair destroys causal information.

The book repeatedly used the sequence:

find first divergence
fix one thing
rerun
allow the next divergence to reveal itself

This is slower per edit and faster per diagnosis.

Observation and intervention are different operations

Instrumentation can change the system.

Examples:

hooks can alter lifetime and add overhead
profiler instrumentation changes timing
.item() can synchronize CUDA
printing values inside compiled code can create graph breaks
a diagnostic optimizer step mutates optimizer state
switching train/eval changes module behavior
seeding changes stochastic execution
gradient clipping changes the gradient you were trying to inspect

Before calling something an observation, ask:

What did this instrument change merely by being present?

If the answer matters, run the final verification again without the instrument.

Known-good references are disproportionately valuable

When an operation is difficult to reason about, reduce the argument.

Use:

manual arithmetic versus PyTorch
manual loss versus F.cross_entropy
manual attention versus scaled_dot_product_attention
eager versus compiled
healthy preprocessing versus candidate preprocessing
healthy run versus candidate run at shared seeds

A reference comparison is strongest when every uncontrolled variable is held fixed.

When the outputs disagree, compare intermediate stages in order.

The first disagreement is usually more useful than the final error magnitude.

The smallest experiment that can distinguish two explanations

A good debugging experiment does not merely make one hypothesis look plausible.

It makes two explanations predict different outcomes.

Examples:

Missing gradient

Hypotheses:

A. graph path is broken
B. path exists but local derivative is zero

Experiment:

clear grad to None
run one backward pass
inspect whether grad remains None or becomes a zero tensor

Low GPU utilization

Hypotheses:

A. input pipeline is starving the device
B. model/host execution is the bottleneck

Experiment:

measure batch wait separately from post-batch work

Compiled slowdown

Hypotheses:

A. steady-state kernel is slower
B. recurring recompilation dominates

Experiment:

measure a repeated fixed-shape steady state
capture recompile logs on the real varying workload

Apparent regression

Hypotheses:

A. model behavior changed
B. ordinary seed variation explains the observation

Experiment:

run both versions on shared seeds and inspect paired deltas

The best experiment is usually not the one with the most instrumentation.

It is the one whose outcomes force the hypotheses apart.

A reusable incident worksheet

Copy this into an issue, notebook or debugging session.

PYTORCH INCIDENT

SYMPTOM
What exactly was observed?

EXPECTED
What specific value, shape, behavior or metric was expected?

REPRODUCTION
What is the smallest input / batch / configuration that reproduces it?

FIRST CONTRACT
Which contract should be checked first?
    tensor
    graph
    ownership
    input representation
    geometry
    attention
    learning chain
    performance
    compiler
    run comparison

OBSERVATIONS
What did the instrument actually report?

FIRST DIVERGENCE
Where does observed behavior first disagree with the expectation?

HYPOTHESES
What are the smallest two or three mechanisms consistent with that evidence?

DISTINGUISHING EXPERIMENT
What one experiment makes those hypotheses predict different outcomes?

INTERVENTION
Change exactly one mechanism.

MUST CHANGE
Which measurement should change if the diagnosis is correct?

MUST NOT CHANGE
Which correctness property must remain unchanged?

VERIFICATION
Did the original failure disappear for the predicted reason?

REGRESSION TEST
What small test should remain so this exact failure cannot silently return?

If you cannot fill in MUST CHANGE and MUST NOT CHANGE, the proposed repair is probably still too vague.

Using AI as a diagnostic partner

The most useful general prompt in this book is not “fix my PyTorch model.”

Use something closer to:

Here is a failing PyTorch system.

Do not rewrite it and do not propose fixes yet.

1. State the expected contract at each relevant boundary.
2. Tell me which facts I can inspect directly and which require a semantic,
   behavioral or provenance probe.
3. Identify the smallest instrument from this list that can localize the failure:
      tensor trace
      autograd reachability
      six-claim audit
      four-structure audit
      producer-consumer timing
      preprocessing stage trace
      CNN shape ledger
      feature-space inspector
      attention four-level check
      learning ledger
      performance ledger
      compiler guard/recompile evidence
      paired run comparison
4. Ask for the missing evidence.
5. Identify the FIRST boundary whose evidence contradicts the intended system.
6. Give at most three mechanisms consistent with that boundary.
7. Propose one experiment that distinguishes those mechanisms.
8. State what should change if the diagnosis is correct.
9. State what must remain unchanged.

Do not propose a repair until the first divergence is supported by evidence.

The purpose of the prompt is not to make the assistant cautious.

It is to make the assistant falsifiable.

When to escalate

Start cheap.

Escalate only when the cheaper instrument passes.

print / assert tensor facts
        ↓
known-sample contract
        ↓
graph / ownership inspection
        ↓
controlled one-step probe
        ↓
tiny-batch capability test
        ↓
hooks / detailed activation inspection
        ↓
profiler / allocator snapshot / compiler logs
        ↓
multi-seed comparison

This is not a universal cost ordering. A seed sweep can be cheap on a tiny workload and a profiler can be cheap on one step.

The principle is:

Use the least invasive instrument that can distinguish the current hypotheses.

When to stop debugging

Stop when all four statements are true:

1. You can name the mechanism that produced the symptom.

2. The repair changed the measurement the mechanism predicted.

3. An independent correctness property remained unchanged.

4. The failure can now be expressed as a test, assertion, contract or recorded
   comparison that would catch it if it returned.

A model that “seems to work now” has not met that standard.

A diagnosis has.

The complete diagnostic stack

The book began with one scalar parameter and ended with a model too large to hold in your head. The instruments accumulated in the same order.

TRAINING LOOP
    forward / backward / update

TENSOR
    first wrong tensor

AUTOGRAD
    first broken dependency edge

RAW NETWORK
    representation / computation / objective /
    differentiation / update / evaluation

MODULE STATE
    Python graph / registered tree /
    autograd graph / optimizer groups

INPUT DELIVERY
    producer / queue / consumer

PREPROCESSING
    structure / numeric interpretation /
    semantics / relationships / provenance

CNN GEOMETRY
    derive / trace / first shape divergence

FEATURE SPACE
    represented object / feature axis /
    quantity / invariance

ATTENTION
    shape / axis semantics /
    numerical invariants / behavior

LEARNING
    TASK → OBJECTIVE → DEPENDENCY → GRADIENT →
    OPTIMIZER OWNERSHIP → PARAMETER UPDATE → CAPABILITY

PERFORMANCE
    measurement contract → phase/resource →
    intervention → uninstrumented remeasurement

COMPILATION
    RUNS → CORRECT → CAPTURED → STABLE → WORTH IT

REGRESSION
    establish the difference →
    verify the measurement →
    localize what changed

CAPSTONE
    choose the instrument that owns the symptom

There is no final universal checker at the bottom of that list.

That is the point.

Different failures violate different contracts, and the dangerous ones often preserve every simpler contract below them.

A tensor can have the right shape and wrong meaning.

An attention block can preserve every external dimension and compare the wrong objects.

A parameter can have a healthy gradient and never move.

A model can minimize the loss and learn the wrong task.

A benchmark can return a precise number for the wrong measurement.

A compiled model can be numerically correct and catastrophically slower.

A new run can be worse without anything meaningful having regressed.

The common skill is not memorizing the fix.

It is knowing what evidence the current symptom does not yet give you.

Make the hidden structure visible. Find the first divergence. Repair the mechanism that caused it. Verify that the system actually improved.