Programmer.ie Book

PyTorch From First Principles

Understand PyTorch from first principles — from tensor geometry and autograd to model structure, transforms, attention, debugging, performance, compilation, reproducibility, and a GPT-style model built from scratch.

PyTorch is easy to start using before it is easy to understand.

You can copy a model, call loss.backward(), create an optimizer, and get a training loop running surprisingly quickly. But that is not the same thing as knowing what the system is doing when the shapes stop lining up, the gradients disappear, the GPU sits idle, torch.compile recompiles unexpectedly, or a model that worked yesterday becomes worse today.

PyTorch From First Principles is designed to close that gap.

This is not an encyclopedic tour of every PyTorch API. It is a practical book about the machinery underneath the APIs: the small number of ideas that let you understand unfamiliar PyTorch code, inspect what it is actually doing, and diagnose it when it goes wrong.

The goal is not to make PyTorch look simple by hiding the difficult parts.

The goal is to make the difficult parts inspectable.

What this book is designed to teach

The book starts with the smallest possible training problem: one parameter, one prediction, one loss, one gradient, one update.

From there, we keep scaling the same ideas upward.

You will learn to think about PyTorch in terms of a few connected structures:

  • tensors and their geometry — shape, dtype, device, stride, broadcasting, layout, and what each dimension actually means;
  • the dynamic computation graph — how operations depend on one another and how autograd follows those dependencies backward;
  • registered model structure — how nn.Module, parameters, buffers, nested modules, state_dict(), device movement, and train/eval behavior fit together;
  • the input contract — how datasets, transforms, batching, augmentation, and preprocessing determine what the model actually sees;
  • the training system — loss, gradients, optimizers, validation, parameter updates, and the evidence that tells you whether learning is really happening;
  • the execution system — CPU/GPU work, memory, profiling, compilation, graph breaks, guards, recompilation, and throughput;
  • the experiment itself — baselines, reproducibility, regressions, benchmark validity, and how to tell whether a change genuinely improved the system.

A recurring rule throughout the book is:

Make hidden structure visible before guessing.

If a tensor fails, inspect its shape and meaning.

If a parameter does not train, inspect registration and gradients.

If a model does not learn, inspect data, targets, loss, gradients, optimizer membership, and parameter movement.

If a GPU is slow, measure where the time is going.

If compilation behaves strangely, inspect graph breaks, guards, and recompiles.

If two experiments differ, establish exactly what changed before explaining the result.

What you should be able to do after reading it

By the end of the book, you should be able to approach a non-trivial PyTorch project and answer questions such as:

What does this tensor represent?
What should its shape be?
What does each dimension mean?
Where did this value come from?
Which tensors belong to the model?
Does the gradient reach this parameter?
Does the optimizer actually own it?
Did the parameter move after the step?
What transformations happened before the model saw this sample?
Why is the GPU waiting?
Where is the memory going?
Why did torch.compile create another graph?
Is this run genuinely worse, or is the comparison itself unreliable?

More concretely, the book is designed to leave you able to:

  • read and debug tensor-heavy PyTorch code instead of reshaping until an error disappears;
  • understand autograd well enough to diagnose missing, stale, exploding, or non-finite gradients;
  • build a neural network with raw tensors before relying on nn.Module;
  • understand why recursive module composition makes large PyTorch models manageable;
  • build and inspect Dataset, DataLoader, and transform pipelines;
  • reason about CNN, embedding, attention, and transformer dimensions;
  • work naturally with feature spaces containing tens, hundreds, or thousands of dimensions;
  • diagnose a model that runs but refuses to learn;
  • profile CUDA execution, memory, and input-pipeline bottlenecks;
  • reason about torch.compile rather than treating it as an unexplained speed switch;
  • design comparable experiments and detect regressions that do not produce exceptions;
  • assemble the mechanisms into a complete small GPT-style language model from scratch.

The final model is important, but it is not the real destination.

The real destination is being able to investigate the model when it does something you did not expect.

How the book gets there

The sequence deliberately introduces mechanisms before convenience abstractions.

Foundations

We begin with training itself, then tensors, autograd, a neural network built without nn.Module, and finally the recursive composition model that explains what nn.Module buys us.

Data and representation

Next we move outside the model and examine how data becomes model input: Dataset, DataLoader, transforms, normalization, augmentation, batching, multiprocessing, and the performance consequences of the input pipeline.

Geometry and architectures

We then apply tensor reasoning to CNNs, high-dimensional feature spaces, an SVM, attention, masks, heads, embeddings, and transformer-style shapes.

The point is not to memorize architecture diagrams. It is to be able to derive the tensor transformations yourself.

Debugging and execution

Once the pieces exist, the book becomes increasingly forensic: models that do not learn, CUDA out-of-memory failures, low GPU utilization, profiler traces, torch.compile, graph breaks, guards, recompilation, and dynamic shapes.

Experiments and regressions

The book then asks a harder question: what if nothing crashes, but the new version is worse?

That leads to baselines, controlled comparisons, reproducibility, benchmark validity, performance regressions, and turning discovered failures into durable tests.

Capstone

Finally, we build a small GPT-style language model from scratch.

By then, embeddings, attention, residual blocks, logits, cross entropy, optimizers, checkpoints, generation, profiling, and debugging should no longer arrive as unrelated pieces of framework magic. They should have somewhere to attach in the mental model you have already built.

Who this book is for

You should be comfortable with basic Python: functions, loops, classes, lists, dictionaries, and running scripts.

You do not need prior PyTorch experience.

You do not need to know transformer internals before starting.

You do not need to understand every part of calculus before using autograd.

The book assumes that you are willing to inspect values, run small experiments, deliberately break code, and ask what changed.

That is more important here than memorizing APIs.

What this book is not trying to do

This book does not attempt to cover every corner of PyTorch.

It does not try to turn distributed training, multi-node infrastructure, custom CUDA kernels, quantization, deployment, serving, LoRA, mixture-of-experts systems, or every current model architecture into mandatory prerequisites for competence.

Those are important topics, but they are specializations.

The purpose of this book is to give them somewhere to attach.

If you understand tensors, computation graphs, registered model structure, data contracts, gradients, optimization, execution, measurement, and debugging, then unfamiliar PyTorch features stop arriving as isolated incantations.

They become extensions of a system you already understand.

The promise

This book will not make you memorize PyTorch.

It is designed to make you understand enough of PyTorch that you can work out what is happening when you encounter something you have not seen before.

That is the standard the rest of the book is built around.

Contents

Chapters