Understand PyTorch from first principles — from tensor geometry and autograd to model structure, transforms, attention, debugging, performance, compilation, reproducibility, and a GPT-style model built from scratch.
PyTorch is easy to start using before it is easy to understand.
You can copy a model, call loss.backward(), create an optimizer, and get a training loop running surprisingly quickly. But that is not the same thing as knowing what the system is doing when the shapes stop lining up, the gradients disappear, the GPU sits idle, torch.compile recompiles unexpectedly, or a model that worked yesterday becomes worse today.
PyTorch From First Principles is designed to close that gap.
This is not an encyclopedic tour of every PyTorch API. It is a practical book about the machinery underneath the APIs: the small number of ideas that let you understand unfamiliar PyTorch code, inspect what it is actually doing, and diagnose it when it goes wrong.
The goal is not to make PyTorch look simple by hiding the difficult parts.
The goal is to make the difficult parts inspectable.
The book starts with the smallest possible training problem: one parameter, one prediction, one loss, one gradient, one update.
From there, we keep scaling the same ideas upward.
You will learn to think about PyTorch in terms of a few connected structures:
nn.Module, parameters, buffers, nested modules, state_dict(), device movement, and train/eval behavior fit together;A recurring rule throughout the book is:
Make hidden structure visible before guessing.
If a tensor fails, inspect its shape and meaning.
If a parameter does not train, inspect registration and gradients.
If a model does not learn, inspect data, targets, loss, gradients, optimizer membership, and parameter movement.
If a GPU is slow, measure where the time is going.
If compilation behaves strangely, inspect graph breaks, guards, and recompiles.
If two experiments differ, establish exactly what changed before explaining the result.
By the end of the book, you should be able to approach a non-trivial PyTorch project and answer questions such as:
What does this tensor represent?
What should its shape be?
What does each dimension mean?
Where did this value come from?
Which tensors belong to the model?
Does the gradient reach this parameter?
Does the optimizer actually own it?
Did the parameter move after the step?
What transformations happened before the model saw this sample?
Why is the GPU waiting?
Where is the memory going?
Why did torch.compile create another graph?
Is this run genuinely worse, or is the comparison itself unreliable?
More concretely, the book is designed to leave you able to:
autograd well enough to diagnose missing, stale, exploding, or non-finite gradients;nn.Module;Dataset, DataLoader, and transform pipelines;torch.compile rather than treating it as an unexplained speed switch;The final model is important, but it is not the real destination.
The real destination is being able to investigate the model when it does something you did not expect.
The sequence deliberately introduces mechanisms before convenience abstractions.
We begin with training itself, then tensors, autograd, a neural network built without nn.Module, and finally the recursive composition model that explains what nn.Module buys us.
Next we move outside the model and examine how data becomes model input: Dataset, DataLoader, transforms, normalization, augmentation, batching, multiprocessing, and the performance consequences of the input pipeline.
We then apply tensor reasoning to CNNs, high-dimensional feature spaces, an SVM, attention, masks, heads, embeddings, and transformer-style shapes.
The point is not to memorize architecture diagrams. It is to be able to derive the tensor transformations yourself.
Once the pieces exist, the book becomes increasingly forensic: models that do not learn, CUDA out-of-memory failures, low GPU utilization, profiler traces, torch.compile, graph breaks, guards, recompilation, and dynamic shapes.
The book then asks a harder question: what if nothing crashes, but the new version is worse?
That leads to baselines, controlled comparisons, reproducibility, benchmark validity, performance regressions, and turning discovered failures into durable tests.
Finally, we build a small GPT-style language model from scratch.
By then, embeddings, attention, residual blocks, logits, cross entropy, optimizers, checkpoints, generation, profiling, and debugging should no longer arrive as unrelated pieces of framework magic. They should have somewhere to attach in the mental model you have already built.
You should be comfortable with basic Python: functions, loops, classes, lists, dictionaries, and running scripts.
You do not need prior PyTorch experience.
You do not need to know transformer internals before starting.
You do not need to understand every part of calculus before using autograd.
The book assumes that you are willing to inspect values, run small experiments, deliberately break code, and ask what changed.
That is more important here than memorizing APIs.
This book does not attempt to cover every corner of PyTorch.
It does not try to turn distributed training, multi-node infrastructure, custom CUDA kernels, quantization, deployment, serving, LoRA, mixture-of-experts systems, or every current model architecture into mandatory prerequisites for competence.
Those are important topics, but they are specializations.
The purpose of this book is to give them somewhere to attach.
If you understand tensors, computation graphs, registered model structure, data contracts, gradients, optimization, execution, measurement, and debugging, then unfamiliar PyTorch features stop arriving as isolated incantations.
They become extensions of a system you already understand.
This book will not make you memorize PyTorch.
It is designed to make you understand enough of PyTorch that you can work out what is happening when you encounter something you have not seen before.
That is the standard the rest of the book is built around.