Example

The loss went to zero and half the model never moved

Run the chapter's two-parameter loop with a detached feature, walk the record PyTorch built, and see which leaves the loss reaches.

While the tensors of a forward pass flow, PyTorch writes down a record of which operation produced which result, with the references needed to differentiate the computation. A missing gradient is a fact about that record.

The chapter’s opening loop detaches a feature before returning it, so the log can hold values without holding the graph. The loss still falls to zero, because the second parameter absorbs the whole job. This program composes the chapter’s pieces into one run: the broken loop, the record, a reachability check and the repair.

Run it

Save this as autograd_record.py and run it with Python 3.11 or later. It needs PyTorch: pip install torch, then python autograd_record.py.

"""Autograd: what the record holds, and where the path to a parameter stops."""
import torch

x = torch.tensor([1.0, 2.0, 3.0, 4.0])
target = torch.tensor([3.0, 6.0, 9.0, 12.0])


def walk(fn, depth=0, leaves=None):
    """Print the record and collect the leaves backward() would reach."""
    if fn is None:
        return leaves if leaves is not None else []
    leaves = [] if leaves is None else leaves
    indent, name = "  " * depth, type(fn).__name__
    if name == "AccumulateGrad":
        leaves.append(fn.variable)
        print(f"{indent}{name} -> leaf tensor, value {fn.variable.item():.4f}")
    else:
        print(f"{indent}{name}")
    for nxt, _ in fn.next_functions:
        walk(nxt, depth + 1, leaves)
    return leaves


def run(detach, steps=400, lr=0.02):
    w1 = torch.tensor(0.5, requires_grad=True)
    w2 = torch.tensor(1.0, requires_grad=True)
    loss = None
    for _ in range(steps):
        f = w1 * x
        prediction = w2 * (f.detach() if detach else f)
        loss = ((prediction - target) ** 2).mean()
        loss.backward()
        with torch.no_grad():
            for p in (w1, w2):
                if p.grad is not None:
                    p -= lr * p.grad
                    p.grad.zero_()
    return w1, w2, loss


w1, w2, loss = run(detach=True)
print("broken loop")
print(f"  loss={loss.item():.6f}  w1={w1.item():.4f}  w2={w2.item():.4f}")
print(f"  w1.grad is None: {w1.grad is None}   w2.grad: {w2.grad}")
print(f"  loss.requires_grad={loss.requires_grad}  grad_fn={type(loss.grad_fn).__name__}")
print("  record of the loss:")
reached = walk(loss.grad_fn)
print(f"  w1 reachable from the loss: {any(t is w1 for t in reached)}")
print(f"  w2 reachable from the loss: {any(t is w2 for t in reached)}")

w1b, w2b, lossb = run(detach=False)
print("repaired loop")
print(f"  loss={lossb.item():.6f}  w1={w1b.item():.4f}  w2={w2b.item():.4f}")
print(f"  w1.grad is None: {w1b.grad is None}")
print(f"  w1 * w2 = {(w1b * w2b).item():.6f}")

The run prints the record of a loss that trained while one parameter received nothing, then the repaired run.

Expected outcome

broken loop
  loss=0.000000  w1=0.5000  w2=6.0000
  w1.grad is None: True   w2.grad: 0.0
  loss.requires_grad=True  grad_fn=MeanBackward0
  record of the loss:
MeanBackward0
  PowBackward0
    SubBackward0
      MulBackward0
        AccumulateGrad -> leaf tensor, value 6.0000
  w1 reachable from the loss: False
  w2 reachable from the loss: True
repaired loop
  loss=0.000000  w1=1.6931  w2=1.7719
  w1.grad is None: False
  w1 * w2 = 3.000001

The record is the evidence

loss.requires_grad is True and loss.grad_fn exists, so backward() has something to start from. That is a statement about the loss, not about w1. The printed record is the statement about w1: the only AccumulateGrad node in the chain belongs to w2. There is no node behind w1, so a gradient had nowhere to be stored.

The chapter’s first diagnostic prints three properties per intermediate tensor; applied to this loop it shows requires_grad going true, false, true again on one line:

def track(name, t):
    print(f"{name:<14} requires_grad={str(t.requires_grad):<5} "
          f"is_leaf={str(t.is_leaf):<5} "
          f"grad_fn={type(t.grad_fn).__name__ if t.grad_fn is not None else 'None'}")
    return t

The reachability check in the program asks the same question from the loss side, before backward() runs. The chapter’s point is that the line where .grad is None and the line where the connection stopped existing are rarely the same line. The break here is the detach() in the helper’s return.

Mechanism and limitations

Documented: the chapter’s opening loop prints w1=0.5000 for four hundred steps while w2 climbs to 6.0000, then shows w1.grad as None. It places the break at the detach().

Observed: the broken loop reached loss 0.000000 with w1 still at 0.5000 and w2 at 6.0000; w1.grad was None while the record held a single AccumulateGrad node, belonging to w2. The reachability check said w1 was not reachable and w2 was. The repaired loop moved w1 to 1.6931, the product to 3.000001.

Not shown. The program composes the chapter’s fragments into one run and merges its graph printer with its reachability walker, so it is not a copy of one listing. The chapter’s walker keeps no reference to the nodes it visits, which can let a recycled object id hide a leaf; this version collects the leaves in a recursive walk instead.

Canonical source , the chapter this example composes. Read the chapter .

Understand this example

Copy this prompt into your AI tool. No code runs here.

Apply this example

Copy this prompt into your AI tool. No code runs here.