PyTorch From First Principles on Programmer.ie

Attention: Which Position Is Comparing With Which?

Fri, 28 Aug 2026 18:00:00 +0100

Here is a tensor of sequence representations and the line that splits it into heads.

B, T, E, Nh = 1, 4, 8, 2
Dh = E // Nh                       # 4

x = torch.arange(B * T * E).reshape(B, T, E)
heads = x.reshape(B, Nh, T, Dh)

x: (1, 4, 8)
heads: (1, 2, 4, 4)

That is exactly the shape multi-head attention wants: batch, heads, positions, head dimension. Nothing raised. Every assertion anyone would think to write passes.

Now the same split written the other way:

Feature Space: What Does a Linear Model Actually See?

Fri, 28 Aug 2026 15:30:00 +0100

Here is a sequence classifier. Eight examples, 128 positions each, 768 features per position, and a linear head that produces one score.

x = torch.randn(8, 128, 768)
classifier = nn.Linear(768, 1)

scores = classifier(x)

input           (8, 128, 768)
Linear(768,1)  -> (8, 128, 1)

One thousand and twenty-four scores where eight were expected. Nothing raised, nothing is non-finite, and the shape is entirely predictable once you know the rule. The layer did exactly what the tensor asked of it: nn.Linear transforms the last axis and preserves every axis before it, so it produced one score for every (example, position) pair — 128 scores per example, computed independently.

CNN Geometry: What Shape Reaches the Next Layer?

Fri, 28 Aug 2026 10:15:00 +0100

Here is a batch of 32 RGB images and an ordinary first convolution.

x = torch.randn(32, 224, 224, 3)
conv = nn.Conv2d(in_channels=3, out_channels=16, kernel_size=3)
conv(x)

RuntimeError: Given groups=1, weight of size [16, 3, 3, 3],
expected input[32, 224, 224, 3] to have 3 channels, but got 224 channels instead

The images have three channels. PyTorch says there are 224. The message names a number that appears nowhere in the model definition, so the obvious reading is that the layer was declared wrong, and the obvious repair is to declare it right:

Transforms: What Does the Model Actually See?

Sat, 22 Aug 2026 11:58:00 +0100

Here are two training runs. Same model, same data, same optimizer, same seed, same number of epochs. The preprocessing pipelines differ by one keyword argument, and neither one raises.

scale=True  input range=[   -2.12,    2.01] first_epoch_loss=   0.0001 last=   0.0000 val_acc=1.000
scale=False input range=[   -2.12,  975.64] first_epoch_loss=  36.7257 last=   1.5337 val_acc=0.620

The second pipeline produced tensors with the right shape, the right dtype, the right device, and no non-finite values. Every check from Chapter 2 passes. Every check from Chapter 5 passes. The batch arrived on time, which is everything Chapter 6 taught us to verify. And the model went from perfect to barely better than guessing.

10: Build a Small GPT-Style Language Model From Scratch

Sat, 08 Aug 2026 14:03:00 +0100

Build a Small GPT-Style Language Model From Scratch in PyTorch

This is the final post in the PyTorch From First Principles series.

We have spent the previous posts learning the machinery underneath PyTorch:

tensors and shapes;
autograd;
manual neural networks;
nn.Module and parameter registration;
DataLoader performance;
convolutional networks;
attention and masks;
training failures;
CUDA performance and torch.compile.

Now we put it together.

The goal is not to download a pretrained model.

Training Regressions and Reproducible Experiments: When Nothing Crashes but the Model Gets Worse

Sat, 08 Aug 2026 14:00:00 +0100

Most bugs in this book have announced themselves loudly.

A tensor had the wrong shape.

A parameter had no gradient.

A model refused to learn.

CUDA ran out of memory.

A compiled graph broke or recompiled.

But some of the most expensive machine-learning bugs are quieter.

The program still runs.

The tests still pass.

Training still finishes.

And yet:

validation loss is worse
training takes 12% longer
peak VRAM is 20% higher
gradients became less stable
compile count increased
throughput dropped
convergence needs twice as many steps

Nothing failed inside the run.

PyTorch Compiler Debugging: Graph Breaks, Guards, Recompiles and torch.compile

Sat, 08 Aug 2026 13:58:00 +0100

In the previous chapter we treated torch.compile as one tool inside a larger performance-debugging workflow.

That is enough until compilation itself becomes the problem.

Then the questions change.

The model runs in eager mode.

The model may even run when compiled.

But perhaps:

compile time is enormous
first call takes seconds
new shapes keep compiling again
graph breaks appear inside forward()
compiled execution is slower than eager
one backend works and another fails
small code changes produce very different compiler behavior

At that point, torch.compile(model) is not the end of the story.

PyTorch Performance Debugging: CUDA OOM, Slow Training, GPU Utilization and torch.compile

Sat, 08 Aug 2026 13:56:00 +0100

At this point in the series, the model runs.

That does not mean it runs well.

A training loop can be correct and still waste most of the machine.

A model can fit in memory and still spend half its time waiting on synchronization.

A torch.compile call can make code faster, slower, or simply move the bottleneck somewhere else.

A CUDA out-of-memory error can be caused by the model, the optimizer, activations, fragmentation, a leaked reference, a larger batch, a longer sequence, or an innocent-looking tensor that was kept alive by Python.

PyTorch Model Not Learning? A Systematic Debugging Guide

Sat, 08 Aug 2026 13:51:00 +0100

Your model runs.

The loss is finite.

Nothing crashes.

And it still does not learn.

This is one of the most frustrating states in machine learning because there is no stack trace telling you what is wrong.

The program is valid Python.

The tensors have legal shapes.

The GPU is busy.

The optimizer is stepping.

And the model is useless.

This post is a systematic way to debug that situation.

DataLoader: Where Is the Training Loop Actually Waiting?

Sat, 08 Aug 2026 13:21:00 +0100

Here are two controlled training pipelines. They use the same batch size, the same machine, and the same synthetic post-batch workload. Each sample also carries the same nominal two-millisecond cost: in one pipeline that cost is waiting, while in the other it is fixed CPU work.

The tensor construction around that controlled cost is the same in both cases. What changes is the resource those two milliseconds consume.

Both are given the same treatment — raise num_workers from 0 to 8 — and measured the same way.

nn.Module: What Does PyTorch Think Belongs to Your Model?

Sat, 08 Aug 2026 13:09:00 +0100

Here is a classifier for the XOR-shaped dataset from Chapter 4, written the way most people write PyTorch. It trains. Validation accuracy climbs from chance to 95.7%. It saves a checkpoint, the checkpoint loads without a warning, and then the reloaded model performs at chance.

import torch
import torch.nn as nn
import torch.nn.functional as F

torch.manual_seed(0)

n = 2000
X = torch.randn(n, 2)
y = ((X[:, 0] > 0) ^ (X[:, 1] > 0)).long()

perm = torch.randperm(n)
X_train, y_train = X[perm[:1600]], y[perm[:1600]]
X_val, y_val = X[perm[1600:]], y[perm[1600:]]

class Encoder(nn.Module):
    def __init__(self, width=16, depth=3):
        super().__init__()
        self.blocks = [nn.Linear(width, width) for _ in range(depth)]

    def forward(self, x):
        for block in self.blocks:
            x = torch.relu(block(x))
        return x

class Classifier(nn.Module):
    def __init__(self, width=16, depth=3):
        super().__init__()
        self.stem = nn.Linear(2, width)
        self.encoder = Encoder(width, depth)
        self.head = nn.Linear(width, 2)

    def forward(self, x):
        x = torch.relu(self.stem(x))
        x = self.encoder(x)
        return self.head(x)

model = Classifier()
optimizer = torch.optim.Adam(model.parameters(), lr=1e-2)

for epoch in range(200):
    logits = model(X_train)
    loss = F.cross_entropy(logits, y_train)

    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

    if epoch % 50 == 0 or epoch == 199:
        with torch.no_grad():
            val_acc = (model(X_val).argmax(1) == y_val).float().mean()
        print(f"epoch={epoch:3d} loss={loss.item():.4f} "
              f"val_acc={val_acc.item():.3f}")

torch.save(model.state_dict(), "classifier.pt")

reloaded = Classifier()
print(reloaded.load_state_dict(torch.load("classifier.pt", weights_only=True)))

with torch.no_grad():
    acc = (reloaded(X_val).argmax(1) == y_val).float().mean()
print(f"reloaded val_acc={acc.item():.3f}")

epoch=  0 loss=0.6964 val_acc=0.512
epoch= 50 loss=0.5930 val_acc=0.810
epoch=100 loss=0.3856 val_acc=0.873
epoch=150 loss=0.2550 val_acc=0.925
epoch=199 loss=0.1908 val_acc=0.957
<All keys matched successfully>
reloaded val_acc=0.512

Read the last two lines together. PyTorch reports that every key in the checkpoint matched every key the model expected, and the model that came back is worthless. Nothing raised. Nothing warned. load_state_dict was as satisfied as it is capable of being.

The Network: What Is It Without nn.Module?

Sat, 08 Aug 2026 12:59:00 +0100

Here is a two-layer neural network, written with nothing but the machinery of the last three chapters. Four trainable tensors, a forward pass, a loss, backward(), an update. It runs without warnings, and the loss falls by a third.

import math
import torch

torch.manual_seed(0)

n = 2000
X = torch.randn(n, 2)
y = ((X[:, 0] > 0) ^ (X[:, 1] > 0)).long()

perm = torch.randperm(n)
n_train = int(0.8 * n)
X_train, y_train = X[perm[:n_train]], y[perm[:n_train]]
X_val,   y_val   = X[perm[n_train:]], y[perm[n_train:]]

D_in, H, C = 2, 16, 2

torch.manual_seed(1)
W1 = (torch.randn(D_in, H) / math.sqrt(D_in)).requires_grad_()
b1 = torch.zeros(H, requires_grad=True)
W2 = (torch.randn(H, C) / math.sqrt(H)).requires_grad_()
b2 = torch.zeros(C, requires_grad=True)
parameters = [W1, b1, W2, b2]

def forward(x):
    z1 = x @ W1 + b1
    return z1 @ W2 + b2

def cross_entropy(logits, targets):
    log_probs = torch.log_softmax(logits, dim=1)
    return -log_probs[torch.arange(targets.shape[0]), targets].mean()

def accuracy(logits, targets):
    return (logits.argmax(dim=1) == targets).float().mean().item()

learning_rate = 0.5

# First verify that one training step is mechanically healthy.
names = ["W1", "b1", "W2", "b2"]
initial_values = [p.detach().clone() for p in parameters]

loss = cross_entropy(forward(X_train), y_train)
before = [p.detach().clone() for p in parameters]

loss.backward()

with torch.no_grad():
    for p in parameters:
        p -= learning_rate * p.grad

for name, p, old in zip(names, parameters, before):
    print(f"{name:3s} leaf={p.is_leaf} grad_none={p.grad is None} "
          f"grad_norm={p.grad.norm().item():8.4f} "
          f"finite={torch.isfinite(p.grad).all().item()} "
          f"moved={(p.detach() - old).abs().max().item():.4f}")

W1  leaf=True grad_none=False grad_norm=  0.4516 finite=True moved=0.1048
b1  leaf=True grad_none=False grad_norm=  0.0192 finite=True moved=0.0053
W2  leaf=True grad_none=False grad_norm=  1.1058 finite=True moved=0.1918
b2  leaf=True grad_none=False grad_norm=  0.0175 finite=True moved=0.0062

Every parameter is a leaf. Every parameter requires gradients. Every parameter receives a finite gradient, and every parameter changes by a measurable amount when the update runs. The basic training machinery is working.

Autograd: What Did PyTorch Record, and Where Does the Gradient Stop?

Sat, 08 Aug 2026 12:55:00 +0100

Here is a training loop with two parameters. It runs, it does not warn, and the loss goes to zero.

import torch

x = torch.tensor([1.0, 2.0, 3.0, 4.0])
target = torch.tensor([3.0, 6.0, 9.0, 12.0])

w1 = torch.tensor(0.5, requires_grad=True)
w2 = torch.tensor(1.0, requires_grad=True)
params = [w1, w2]

feature_log = []

def stage_one(t):
    f = w1 * t
    detached = f.detach()          # keep the values for later inspection
    feature_log.append(detached)
    return detached

learning_rate = 0.02

for step in range(400):
    f = stage_one(x)
    prediction = w2 * f
    loss = ((prediction - target) ** 2).mean()

    loss.backward()

    with torch.no_grad():
        for p in params:
            if p.grad is not None:
                p -= learning_rate * p.grad
                p.grad.zero_()

    if step in (0, 9, 99, 399):
        print(f"step={step:3d} loss={loss.item():.6f} "
              f"w1={w1.item():.4f} w2={w2.item():.4f}")

step=  0 loss=46.875000 w1=0.5000 w2=1.3750
step=  9 loss=11.521060 w1=0.5000 w2=3.7071
step= 99 loss=0.000009 w1=0.5000 w2=5.9979
step=399 loss=0.000000 w1=0.5000 w2=6.0000

Read the columns the way Chapter 1 taught. The loss falls to zero, which is the outcome we asked for. And w1 never moves. Not slowly, not by a little: it holds 0.5000 for four hundred steps while w2 climbs to exactly 6.0, which is the value that makes w1 * w2 equal to the 3 we were trying to learn. Half the model was frozen, the other half absorbed the entire job, and the objective was satisfied anyway.

The Tensor: What Is Actually Flowing Through the Loop?

Sat, 08 Aug 2026 12:45:00 +0100

The last program in Chapter 1 trained a two-parameter model on four examples. It worked, and we read it as a story about gradients: predict, measure, differentiate, step. But something in it went unexamined. w was a single number and x held four, and w * x + b produced four predictions without anyone specifying how a scalar and a four-element vector should combine. PyTorch had a rule. We never stated it.

What Are We Actually Doing?

Sat, 08 Aug 2026 12:40:00 +0100

It has never been easier to obtain working PyTorch code. Describe a model to an assistant, paste the result into a file, and a training loop will usually appear and usually run. For a large fraction of everyday work this is fine, and this book does not ask you to pretend otherwise.

The difficulty arrives afterwards.

The code runs, but the loss sits at exactly the same value for two hundred steps. Or one tensor is [32, 128] where the next operation wanted [128, 32]. Or the gradient of a weight is None and nothing says why. Or the model trained yesterday and today it does not, with no change you can point to. None of these are syntax errors. They do not produce a helpful traceback, and often they produce no traceback at all. They are questions about what the program is doing, and to answer them you need somewhere to look.

Mon, 01 Jan 0001 00:00:00 +0000

PyTorch From First Principles — Working Ledger

Book intention

Learn PyTorch by understanding what the framework is doing from first principles, then build upward from the core training loop to modern model architectures.

This book is already substantially written. The ledger does not assume that published chapters are editorially complete: the new pass should test the progression, remove avoidable duplication, and preserve the practical debugging character that makes the sequence useful.