The Tensor: What Is Actually Flowing Through the Loop?

The last program in Chapter 1 trained a two-parameter model on four examples. It worked, and we read it as a story about gradients: predict, measure, differentiate, step. But something in it went unexamined. w was a single number and x held four, and w * x + b produced four predictions without anyone specifying how a scalar and a four-element vector should combine. PyTorch had a rule. We never stated it.

Here is why that matters. Take that same loop and change one thing. The targets now arrive as a column rather than a row, which is what happens when they come out of a spreadsheet column, a DataLoader, or a model that emits one value per example:

import torch

x = torch.tensor([1.0, 2.0, 3.0, 4.0])
target = torch.tensor([[3.0], [6.0], [9.0], [12.0]])     # was [3.0, 6.0, 9.0, 12.0]

w = torch.tensor(1.0, requires_grad=True)
b = torch.tensor(0.0, requires_grad=True)

for step in range(500):
    prediction = w * x + b
    loss = ((prediction - target) ** 2).mean()

    loss.backward()

    with torch.no_grad():
        w -= 0.05 * w.grad
        b -= 0.05 * b.grad

    w.grad.zero_()
    b.grad.zero_()

    if step in (0, 9, 99, 499):
        print(f"step={step:3d} loss={loss.item():.6f} w={w.item():.4f} b={b.item():.4f}")

step=  0 loss=37.500000 w=2.1250 b=0.5000
step=  9 loss=17.536465 w=2.0555 b=1.4566
step= 99 loss=11.666244 w=0.5289 b=5.9449
step=499 loss=11.250002 w=0.0013 b=7.4963

Nothing raised. The loss fell by two thirds and then settled, which is roughly what a converging run looks like. And the model is useless: w went to zero instead of three, so the answer is y = 7.5 regardless of the input.

Chapter 1 taught us to check the three phases when a model does not learn. Every one of them is fine here. The forward pass computes what it was told to compute, backward() populates both gradients, and the update moves both parameters. The bug is not in the loop. It is in an object that flowed through the loop having a different shape from the one we assumed, and an operation that accommodated the difference instead of complaining.

This chapter is about those objects. The goal is a specific and checkable skill: given a tensor operation, work out what it will produce before you run it, and when reality disagrees, find the first place it diverged. That skill is what makes the rest of the book possible. Convolutions, embeddings, attention and transformers are, mechanically, sequences of shape transformations, and a great deal of the work of debugging them is deciding which transformation first produced something you did not intend.

We will come back to the loop above and account for the number 7.5 exactly.

A tensor is a description, not just a container

torch.tensor(2.0) looks like a box holding a number, and while everything was a scalar that reading cost us nothing. It is not sufficient now. A tensor carries a block of values plus a description of how to interpret them, and every part of that description corresponds to a class of failure you will eventually meet.

import torch

def describe(name, t):
    print(f"{name:<8} shape={tuple(t.shape)!s:<16} dtype={str(t.dtype):<14}"
          f" device={t.device} stride={t.stride()} contiguous={t.is_contiguous()}")

x = torch.randn(8, 16, 64)

describe("x", x)
describe("x.T2", x.transpose(1, 2))

x        shape=(8, 16, 64)      dtype=torch.float32  device=cpu stride=(1024, 64, 1) contiguous=True
x.T2     shape=(8, 64, 16)      dtype=torch.float32  device=cpu stride=(1024, 1, 64) contiguous=False

Five properties, five kinds of question:

shape is the logical extent along each axis, and ndim is how many axes there are. This is what most operations validate, and it is what most tracebacks report.

dtype is the element type. It governs whether an operation is even defined, how much precision survives it, and how two operands combine.

device is where the memory lives. Operations generally require their operands to be in the same place.

stride says how many elements to step in storage to move one position along each axis. It is how PyTorch converts a logical index into an offset, and it explains why the transposed tensor above has the same values in the same memory while reporting a different shape.

is_contiguous() reports whether the current strides correspond to the standard row-major layout for the current shape.

Shape and dtype describe how the values are logically interpreted; device says where those values live; strides and contiguity describe how logical positions map onto storage.

Keep those layers separate. A shape problem, a dtype problem, a device problem and a layout problem can produce very different symptoms and require very different repairs.

The describe helper is worth keeping. Most of the diagnostic work in this chapter is a matter of having looked at these values before forming an opinion.

Shape carries a semantic contract PyTorch cannot enforce

Consider two tensors:

a = torch.randn(32, 128)
b = torch.randn(128, 32)

They hold the same number of values, and Python’s type system considers both to be torch.Tensor. Many operations are perfectly legal on either one. But if the first means 32 examples with 128 features each and the second reverses those axes, the same legal operation can have a completely different meaning. PyTorch has no way to know that. It stores the integers 32 and 128. The words batch and features exist only in your head and in the code you wrote around the tensor.

So write them down. The convention this book uses throughout is a short symbolic name per axis:

(B, F)        batch, features
(B, T, D)     batch, sequence position, embedding dimension
(B, C, H, W)  batch, channel, image height, image width
(B, H, T, Dh) batch, attention head, sequence position, per-head dimension

The letters are not PyTorch syntax and they have no runtime effect. They are a discipline for reading. When you encounter torch.randn(32, 100, 768), the useful reflex is not to note that it has three dimensions but to ask three questions: 32 what? 100 what? 768 what? If you cannot answer, you do not yet know what the code does, and neither will an assistant you ask about it.

This gives us a distinction that runs through the rest of the book. An operation can fail in two very different ways.

Structurally illegal. PyTorch cannot perform the operation you asked for, and says so. You get a traceback.

Structurally legal but semantically wrong. PyTorch performs exactly the operation you asked for. You asked for the wrong one. There is no traceback, and the program produces numbers.

The first kind is usually easier because execution stops and gives you evidence. The second can survive long enough to look like a training problem rather than a tensor problem.

There is a special case worth naming now, because it will bite you during debugging. When two axes happen to have the same size, the shape carries no information at all about which is which:

h = torch.randn(32, 32)

Is that (batch, hidden) or (hidden, batch)? Both. Neither. Nothing in the tensor distinguishes them, no assertion on h.shape can distinguish them, and a transposition somewhere upstream would leave this tensor looking perfectly correct. It follows that when you are reproducing a suspected shape bug, you should choose sizes that are all different from one another. A batch of 32 flowing into a hidden width of 32 is a debugging environment that has been designed to hide the bug. We will use this deliberately later in the chapter.

What an operation can do to a shape

Deriving output shapes does not require memorizing the signature of every function in PyTorch. A surprisingly large fraction of everyday tensor operations can be understood through a small set of shape transformations, each with a rule you can apply on paper.

Preserve it. Elementwise unary operations return the same shape they were given. torch.relu(x), x.exp(), x * 2, x.float(). If a shape changed, one of these was not the culprit.

Align two shapes and expand. Elementwise binary operations broadcast. Line the shapes up from the right; each pair of dimensions must be equal, or one of them must be 1; missing leading dimensions are treated as 1. The output takes the larger size in each position.

(32, 10, 128)          (32, 10, 128)          (32, 10, 128)
        (128)   ->  ok      (10, 128) ->  ok    (1, 1, 128) -> ok

(32, 10, 128)
     (32, 128)   ->  128 matches 128, but 10 and 32 do not: error

You can ask PyTorch to do this arithmetic for you without allocating anything:

print(torch.broadcast_shapes((32, 1), (32,)))

torch.Size([32, 32])

Contract an axis. Reductions consume one or more axes. sum, mean, max, norm and friends remove the axes named in dim, unless keepdim=True leaves them behind with size 1. With no dim, everything is reduced to a scalar with ndim == 0.

x = torch.randn(4, 6, 8)

print(x.sum(dim=1).shape)
print(x.sum(dim=1, keepdim=True).shape)
print(x.sum(dim=(0, 2)).shape)
print(x.sum().shape, x.sum().ndim)

torch.Size([4, 8])
torch.Size([4, 1, 8])
torch.Size([6])
torch.Size([]) 0

Matrix multiplication is also a contraction: it consumes the matching inner dimension.

(..., M, K) @ (..., K, N)  ->  (..., M, N)

The leading dimensions broadcast against each other by the same rule as above. There are two special cases for one-dimensional operands that are worth committing to memory, because they are a common source of a shape you did not expect:

print((torch.randn(3) @ torch.randn(3)).shape)          # both 1-D: inner product
print((torch.randn(5, 3) @ torch.randn(3)).shape)       # 1-D on the right
print((torch.randn(3) @ torch.randn(3, 5)).shape)       # 1-D on the left
print((torch.randn(2, 5, 3) @ torch.randn(3, 4)).shape) # batched against a matrix
print((torch.randn(2, 1, 5, 3) @ torch.randn(3, 3, 4)).shape)

torch.Size([])
torch.Size([5])
torch.Size([3, 5])
torch.Size([2, 5, 4])
torch.Size([2, 3, 5, 4])

For matmul, a one-dimensional left operand is treated as though a leading dimension of size 1 had been added, while a one-dimensional right operand is treated as though a trailing dimension of size 1 had been added. After multiplication, that temporary dimension is removed from the result. So (5, 3) @ (3,) gives (5,), not (5, 1). If you were expecting a column, you have just silently lost an axis.

Insert or remove an axis. Integer indexing removes the axis it indexes; slicing keeps it. unsqueeze(d) inserts an axis of size 1; squeeze(d) removes one if it has size 1, and does nothing if it does not.

im = torch.randn(4, 3, 32, 32)

print(im[0].shape)        # integer index: axis removed
print(im[0:1].shape)      # slice: axis kept
print(im[:, 0].shape)
print(im[..., 0].shape)

torch.Size([3, 32, 32])
torch.Size([1, 3, 32, 32])
torch.Size([4, 32, 32])
torch.Size([4, 3, 32])

This is behind a failure mode you should recognize on sight: code that works with a batch of 8 and fails with a single example. Passing batch[0] hands the next function a (C, H, W) tensor where it expected (B, C, H, W). Passing batch[0:1] preserves the contract.

squeeze() with no argument deserves particular suspicion, because it removes every size-one axis, including ones you needed:

s = torch.randn(1, 10, 1)

print(s.squeeze().shape)
print(s.squeeze(-1).shape)
print(s.squeeze(0).shape)

torch.Size([10])
torch.Size([1, 10])
torch.Size([10, 1])

The first line destroyed the batch axis. In a pipeline where batch size is usually greater than one, that is a bug which appears only on the last incomplete batch, or only during single-example inference. Name the axis you mean.

Reinterpret the same elements. reshape, view and flatten keep every value and change how the axes divide them. The element count is conserved, which is the one constraint they enforce, and -1 asks PyTorch to solve for a single unknown axis.

x = torch.randn(32, 3, 28, 28)

print(x.reshape(x.shape[0], -1).shape)
print(torch.flatten(x, start_dim=1).shape)

torch.Size([32, 2352])
torch.Size([32, 2352])

Both produce the same thing here, and flatten(x, start_dim=1) says what it means: keep the batch axis, collapse everything after it. Prefer x.shape[0] over a hard-coded 32, so that the code does not silently depend on a batch size that will eventually change.

Reorder the axes. transpose(a, b) swaps two; permute(...) specifies the whole new order. The values and the element count are unchanged; only which axis is which.

images = torch.randn(32, 224, 224, 3)     # (B, H, W, C)
images = images.permute(0, 3, 1, 2)       # (B, C, H, W)

print(images.shape)

torch.Size([32, 3, 224, 224])

This list is a working checklist rather than a closed taxonomy. Joining operations such as cat and stack extend or add an axis, padding grows one, and convolutions compute new spatial extents from kernel and stride parameters, which Chapter 8 takes apart. But when you meet an unfamiliar operation, asking which of these it is will usually get you to a prediction, and a prediction is the thing you can check.

Two of these categories are particularly important for debugging because both can succeed while doing something other than what you meant. We take them in turn.

Broadcasting does exactly what you asked

Broadcasting is what let us write w * x + b in Chapter 1 without thinking about it, and it is the reason a bias vector can be added to a batch of activations without anyone materializing one copy per row. It is not a dangerous feature. It is a feature that is willing to make sense of shape combinations you did not intend to write.

Here is the canonical version:

torch.manual_seed(0)

predictions = torch.randn(32, 1)      # one output per example
targets = torch.randn(32)             # one target per example

diff = predictions - targets

print(diff.shape, diff.numel())

torch.Size([32, 32]) 1024

Thirty-two predictions, thirty-two targets, and one thousand and twenty-four differences.

Work through why PyTorch considers this reasonable. Align the shapes from the right:

predictions   (32, 1)
targets           (32)

The rightmost pair is 1 against 32. A 1 is allowed to expand, so it becomes 32. Moving left, predictions has a 32 and targets has nothing, so the missing dimension is treated as 1 and expands too. The result is (32, 32), and element [i, j] is predictions[i] - targets[j]. Every prediction has been compared against every target. The operation is exactly what the rules specify. The rules were applied to shapes that did not mean what we assumed.

Now watch how well this bug hides:

wrong = (diff ** 2).mean()
right = ((predictions.squeeze(1) - targets) ** 2).mean()

print(f"wrong={wrong.item():.4f}  right={right.item():.4f}")

wrong=2.1872  right=2.3178

The wrong loss is a plausible number. It is not NaN, not enormous, not zero, and in this case it is slightly smaller than the correct loss, so a run using it looks marginally better rather than obviously broken. It is a scalar, so backward() works on it. Gradients flow to predictions with the right shape. Every downstream check you might casually apply passes.

The evidence that exposes it is the shape itself, and specifically the element count. Thirty-two examples should produce thirty-two errors. numel() reporting 1024 is the fact that does not fit any correct story.

Back to the loop

Now we can account for 7.5 completely. In the program that opened this chapter, prediction had shape (4,) and target had shape (4, 1). Align them:

prediction       (4)
target       (4, 1)

The rightmost pair is 4 against 1, so the 1 expands. The missing leading dimension of prediction expands too. The difference is (4, 4), and .mean() averages sixteen values rather than four. The objective being minimized is not “the average squared error of each prediction against its own target”. It is “the average squared error of every prediction against every target”.

That objective has an analysable minimum, which means we can predict what training will converge to instead of guessing. For a fixed set of targets, the value of a single prediction p that minimizes the average of (p - t_j)² over all targets t_j is the mean of the targets. The pairing has been destroyed, so no prediction has any reason to differ from any other, and the best each can do is sit at the target mean. Our targets are 3, 6, 9, 12, whose mean is 7.5. The residual loss at that point is the mean squared deviation of the targets from their own mean, which is 11.25.

Both numbers appear in the output: b converged to 7.4963 with w at 0.0013, and the loss settled at 11.250002. That agreement is the point. We did not diagnose this by trying fixes until the run looked better. We formed a hypothesis about which operation had the wrong shape, derived a consequence that could only be true if the hypothesis was right, and checked it. This is the same move as Chapter 1’s -19.2 = -3.2 + (-16.0).

The repair is to state the contract rather than to reshape until the numbers change:

assert prediction.shape == target.shape, (
    f"prediction {tuple(prediction.shape)} vs target {tuple(target.shape)}"
)

Placed just before the loss, that assertion fails immediately, in the loop body, naming both shapes. Whether you then fix it by squeezing the target or by giving the prediction a trailing axis depends on which shape the rest of the program expects. What matters is that you decided, rather than letting the broadcasting rules decide for you.

And verify the repair against something independent. Loss going down is not the check; we already know the broken version’s loss goes down. The check is that w converges to 3.0, b to 0.0, and the held-out prediction at x = 10 approaches 30.0, exactly as it did in Chapter 1.

Broadcasting failures do not announce themselves, because from PyTorch’s point of view nothing failed. The signal is a shape or an element count larger than the quantity you believe you are computing.

Reductions and the axis you did not mean

The second category that succeeds while doing the wrong thing is reduction, and standardizing a batch of features is the clearest example.

x = torch.randn(32, 64)      # (B, F)

a = (x - x.mean(dim=0)) / x.std(dim=0)
b = (x - x.mean(dim=1, keepdim=True)) / x.std(dim=1, keepdim=True)

print(a.shape, b.shape)

torch.Size([32, 64]) torch.Size([32, 64])

Two lines, same input, same output shape, no error from either. They compute different things. The first standardizes each feature using statistics gathered across the batch, which is what “normalize the inputs” usually means. The second standardizes each example using statistics gathered across its own features, which is a different transformation with different consequences.

Because the shapes match, no assertion on shape will separate them. The discriminating evidence is a property of the result rather than its shape: ask which axis now has zero mean.

print(f"a: max |mean| over dim 0 = {a.mean(dim=0).abs().max():.2e}")
print(f"b: max |mean| over dim 0 = {b.mean(dim=0).abs().max():.2e}")
print(f"b: max |mean| over dim 1 = {b.mean(dim=1).abs().max():.2e}")

a: max |mean| over dim 0 = 4.47e-08
b: max |mean| over dim 0 = 5.04e-01
b: max |mean| over dim 1 = 5.40e-08

That is a general technique and worth extracting from the example. When two candidate implementations produce the same shape, look for a scalar summary that the two hypotheses predict differently, and compute it. Here, “the features are standardized” and “the examples are standardized” make incompatible predictions about which marginal mean is zero, and one measurement settles it.

keepdim deserves a note of its own, because it interacts with broadcasting in a way that produces both kinds of failure depending on the sizes involved. Without it, x.mean(dim=1) on a (32, 64) tensor gives (32,), and subtracting that from (32, 64) aligns 32 against 64 and raises. Helpful. But on a square tensor:

sq = torch.randn(32, 32)

print((sq - sq.mean(dim=1)).shape)
print(torch.equal(sq - sq.mean(dim=1), sq - sq.mean(dim=1, keepdim=True)))

torch.Size([32, 32])
False

Legal, correctly shaped, and different. Without keepdim, the row means became a (32,) vector which broadcast along the last axis, so each column was centered by the mean of the correspondingly-indexed row. This is the square-tensor blind spot from earlier, arriving in real code.

Find the first wrong tensor, not the first illegal one

Everything so far has been about a single operation. Real code is a chain of them, and the practical difficulty is that the operation which raises is often not the operation that was wrong.

Here is a small forward pass written by hand. It follows the convention nn.Linear uses internally, where a weight is stored as (out_features, in_features); Chapter 4 builds a network this way from scratch.

torch.manual_seed(0)

in_features, hidden, classes = 64, 32, 10

W1 = torch.randn(hidden, in_features)      # (32, 64)
W2 = torch.randn(classes, hidden)          # (10, 32)

def forward(x):                            # x: (B, in_features)
    h = W1 @ x.T
    h = torch.relu(h)
    logits = h @ W2.T
    return logits

x = torch.randn(32, in_features)
print(forward(x).shape)

torch.Size([32, 10])

Thirty-two examples in, thirty-two rows of ten class scores out. It looks right.

If W1 and W2 were trainable parameters inside a model, this dimensional mistake could survive into training at this particular batch size because every operation remains structurally legal. The important point here is not what optimization would eventually do with it; it is that the shape trace alone currently looks correct.

Then someone runs the last incomplete batch of an epoch, or halves the batch size to fit in memory:

forward(torch.randn(16, in_features))

RuntimeError: mat1 and mat2 shapes cannot be multiplied (32x16 and 32x10)

The traceback points at h @ W2.T, the last line of the function. The instinct at this point is to make that line work: transpose W2, or insert a reshape, or flatten something. Any of those can be made to stop the exception. None of them addresses what happened.

Instead, state the contract and find where it breaks. Every tensor in this function has an expected symbolic shape, and we know what those are because we wrote the function:

x       (B, in_features)
h       (B, hidden)
relu    (B, hidden)
logits  (B, classes)

Instrument it. This does not need to be clever:

def check(name, t, expected):
    actual = tuple(t.shape)
    mark = "ok      " if actual == expected else "DIVERGES"
    print(f"{mark} {name:<7} expected={str(expected):<12} actual={actual}")
    return t

def forward(x, B):
    check("x", x, (B, in_features))
    h = W1 @ x.T
    check("h", h, (B, hidden))
    h = torch.relu(h)
    check("relu", h, (B, hidden))
    logits = h @ W2.T
    check("logits", logits, (B, classes))
    return logits

Run it at the batch size that failed:

forward(torch.randn(16, in_features), 16)

ok       x       expected=(16, 64)     actual=(16, 64)
DIVERGES h       expected=(16, 32)     actual=(32, 16)
DIVERGES relu    expected=(16, 32)     actual=(32, 16)
RuntimeError: mat1 and mat2 shapes cannot be multiplied (32x16 and 32x10)

The first divergence is at h, two operations before the exception. relu is downstream damage: an elementwise operation preserves shape, so it could only ever have propagated what it was given. And the shape it reports, (32, 16), is (hidden, B). The axes are the ones we wanted, in the wrong order.

Which identifies the mechanism exactly. W1 @ x.T is (hidden, in_features) @ (in_features, B), giving (hidden, B). The intended operation puts the batch first: x @ W1.T, which is (B, in_features) @ (in_features, hidden), giving (B, hidden). Somebody had a weight stored as (out, in), correctly recognized that a transpose was needed, and applied it to the wrong operand.

def forward(x):
    h = x @ W1.T
    h = torch.relu(h)
    return h @ W2.T

Verify at more than one batch size, which is the whole point:

print(forward(torch.randn(32, in_features)).shape)
print(forward(torch.randn(16, in_features)).shape)

torch.Size([32, 10])
torch.Size([16, 10])

Now the part that matters most. Run the original instrumented function at batch size 32:

ok       x       expected=(32, 64)     actual=(32, 64)
ok       h       expected=(32, 32)     actual=(32, 32)
ok       relu    expected=(32, 32)     actual=(32, 32)
ok       logits  expected=(32, 10)     actual=(32, 10)

Every check passes. The output shape is (32, 10), exactly as specified. And the values are wrong, because (hidden, B) and (B, hidden) are indistinguishable when hidden and B are both 32. Confirm it directly:

def broken(x):
    return torch.relu(W1 @ x.T) @ W2.T

def fixed(x):
    return torch.relu(x @ W1.T) @ W2.T

x = torch.randn(32, in_features)
print(torch.allclose(broken(x), fixed(x)))

False

The exception at batch size 16 was not the bug appearing. It was the bug becoming visible, because a coincidence between two axis sizes stopped holding. The bug had been there, quietly transposing a batch of activations, for as long as batch size happened to equal hidden width.

Two things to carry away from this. The first is a procedure:

State the expected symbolic shape of each tensor. Compare against the observed shape. Find the earliest disagreement. Investigate the operation immediately before it. Only then decide what to change.

The second is a habit that makes the procedure work. Choose sizes that differ from each other when you are testing tensor code. A batch of 32 into a hidden width of 32 is a configuration in which an entire class of bug is undetectable. Use 16 and 32, or 7 and 13. If a shape bug survives that, it is a real one rather than an arithmetic coincidence.

Shape assertions are worth writing, and we will formalize them later in the chapter, but this example bounds what they can do. An assertion compares integers. It cannot tell you that the integers mean the wrong thing.

Shape describes interpretation; strides describe storage

So far every question has been about the logical shape. There is a second layer underneath, and it explains a family of errors that otherwise look arbitrary, in particular the one that arrives when a view() refuses after a transpose().

A tensor’s values live in a flat, one-dimensional region of storage. The shape says how many positions exist along each axis. The strides say how far to step through storage to move one position along each axis. Together they turn a logical index into an offset.

x = torch.arange(12).reshape(3, 4)

print(x.stride(), x.is_contiguous())

(4, 1) True

Moving one step along axis 1 moves one element in storage; moving one step along axis 0 skips four. That is exactly what you would expect for a row-major matrix with four columns, and it is what is_contiguous() reports: the strides match the standard layout for this shape.

Now transpose it:

y = x.t()

print(y.shape, y.stride(), y.is_contiguous())
print("same storage:", y.data_ptr() == x.data_ptr())

torch.Size([4, 3]) (1, 4) False

same storage: True

Nothing was copied. data_ptr() confirms that these two tensors begin at the same address; PyTorch produced a new view of the same storage, with the strides swapped to match the swapped axes.

Creating that transposed view is cheap because no tensor data is moved. The resulting non-contiguous layout can still matter to operations that consume it later. It is also why the result is not contiguous: the standard layout for a (4, 3) tensor would be strides (3, 1), and these are (1, 4).

That is the whole mechanism, and the view() error follows from it directly:

y.view(-1)

RuntimeError: view size is not compatible with input tensor's size and stride
(at least one dimension spans across two contiguous subspaces). Use .reshape(...) instead.

view() promises to return a tensor sharing the original storage. Flattening y in row-major order means reading 0, 4, 8, 1, 5, 9, ..., and no single stride can express that walk through the existing storage. PyTorch cannot honor the promise, so it declines rather than silently copying.

reshape() makes a weaker promise: give me this shape, sharing storage if that is possible and copying if it is not.

print(y.reshape(-1).tolist())
print("copied:", y.reshape(-1).data_ptr() != x.data_ptr())
print("copied:", x.reshape(-1).data_ptr() != x.data_ptr())

[0, 4, 8, 1, 5, 9, 2, 6, 10, 3, 7, 11]
copied: True
copied: False

Same call, two different behaviors, decided by layout. reshape on the transposed tensor had to allocate; on the original contiguous tensor it returned a view.

This is where the rules people repeat to each other go wrong, so state the mechanism precisely rather than a slogan.

It is not true that view() requires a contiguous tensor. It requires that the requested shape be expressible with strides over the existing storage:

g = torch.randn(4, 6)
h = g[:, :3]                     # a slice: non-contiguous

print(h.stride(), h.is_contiguous())
print(h.view(4, 3, 1).shape)     # works, on a non-contiguous tensor

(6, 1) False
torch.Size([4, 3, 1])

It is not true that view() never works after permute(). If the permutation happens to leave the result in a layout compatible with the request, it works:

c = torch.randn(1, 6)
d = c.permute(1, 0)

print(d.shape, d.stride(), d.is_contiguous())
print(d.view(6).shape)

torch.Size([6, 1]) (1, 6) True
torch.Size([6])

Permuting axes of size one rearranges the description without disturbing the walk through storage, so the result is still contiguous.

And it is not true that reshape() always copies, as the data_ptr() comparison above showed. What is true is narrower and more useful:

view() returns a tensor sharing storage or raises. reshape() returns the requested shape, sharing storage when the layout permits and copying otherwise. Whether a given call copies depends on the tensor’s strides, so do not write code whose correctness depends on which one happened.

That last clause is a real constraint, because sharing storage means aliasing:

a = torch.zeros(2, 3)
v = a.view(6)
v[0] = 99

print(a)

tensor([[99.,  0.,  0.],
        [ 0.,  0.,  0.]])

Writing through a view modified the original. If the same code path had produced a copy, because some upstream operation left the tensor non-contiguous, the write would have gone nowhere visible and the original would still be zero. That is a genuinely unpleasant bug to chase, and the way to avoid it is not to combine in-place writes with reshaping whose return type you have not established.

Finally, contiguous() ensures a contiguous layout. If the tensor is already contiguous, PyTorch can return it unchanged; if it is not, producing a contiguous result requires copying the data:

y = x.t().contiguous()
print(y.is_contiguous(), y.view(-1).shape)

True torch.Size([12])

Reaching for .contiguous() the moment a view() complains will make the error go away. Before doing that, it is worth knowing which earlier operation made the tensor non-contiguous, because occasionally the answer is that the operation itself was a mistake, and you are about to pay for a copy in order to preserve it.

Same shape, different tensor

We can now put the two layers together in the sharpest example in this chapter. Take a batch of token embeddings and split the embedding dimension across attention heads. Chapter 10 is about what this is for; here it is a shape exercise.

(B, T, D)  ->  (B, H, T, Dh)      where D = H * Dh

Two implementations. Both produce the target shape. Both run.

B, T, H, Dh = 1, 4, 2, 3
D = H * Dh

x = torch.arange(B * T * D).reshape(B, T, D)

good = x.reshape(B, T, H, Dh).transpose(1, 2)
bad = x.reshape(B, H, T, Dh)

print(good.shape, bad.shape, torch.equal(good, bad))

torch.Size([1, 2, 4, 3]) torch.Size([1, 2, 4, 3]) False

Identical shapes, different contents. Print the first head of each, with x alongside so you can see where the values came from:

print(x[0])
print(good[0, 0])
print(bad[0, 0])

tensor([[ 0,  1,  2,  3,  4,  5],
        [ 6,  7,  8,  9, 10, 11],
        [12, 13, 14, 15, 16, 17],
        [18, 19, 20, 21, 22, 23]])

tensor([[ 0,  1,  2],
        [ 6,  7,  8],
        [12, 13, 14],
        [18, 19, 20]])

tensor([[ 0,  1,  2],
        [ 3,  4,  5],
        [ 6,  7,  8],
        [ 9, 10, 11]])

Each row of x is one token; each column is one embedding dimension. Head 0 should own the first three embedding dimensions of every token, and that is what good contains: columns 0 to 2 of all four rows.

bad contains the same values, but they have been grouped according to the wrong axes. Instead of splitting each token’s embedding dimension across heads, the reshape partitions contiguous chunks of the flattened sequence-and-embedding region.

In this small example, that makes head 0 consume all the values from the first two tokens, rearranged into four rows of three. Downstream operations can remain perfectly legal because the final shape is still (B, H, T, Dh). They will simply be operating on a different interpretation of the data from the one we intended.

The reason is that reshape operates on the flat ordering of the elements, and in (B, T, D) the embedding is the fastest-varying axis. Splitting the last axis into (H, Dh) respects that ordering, which is why reshape(B, T, H, Dh) is correct and a subsequent transpose(1, 2) is needed to move H forward. Writing reshape(B, H, T, Dh) asks for a different division of the same flat sequence, and gets it.

The layout consequence follows immediately. transpose produced a non-contiguous tensor, so:

good.view(B, H, T * Dh)

RuntimeError: view size is not compatible with input tensor's size and stride ...

while good.reshape(B, H, T * Dh) succeeds by copying. That error is not a sign that the head split was wrong. It is the expected consequence of a correct transpose, and the fix is reshape or an explicit contiguous(), chosen knowing that a copy occurs.

Two operations that produce the same shape are not therefore the same operation. When a reshape and a permutation can both reach the target shape, work out which elements end up where, ideally on a small tensor of arange values where you can read the answer off the screen.

That last suggestion is the most portable debugging technique in this chapter. Random data tells you nothing about where values moved. Sequential integers tell you everything, and a tensor small enough to print will settle in ten seconds a question you could otherwise argue about for an hour.

dtype and device are part of the contract

Shape is one major source of structural surprises, but it is not the only property an operation cares about. dtype and device produce their own characteristic failures.

Integer tensors are the first thing to watch, because Python’s literals and PyTorch’s defaults do not always agree with your intentions:

i = torch.tensor([1, 2, 3])
print(i.dtype)
i.mean()

torch.int64

RuntimeError: mean(): could not infer output dtype. Input dtype must be either
a floating point or complex dtype. Got: Long

That one is loud, which is the good case. This one is not:

a = torch.tensor([200], dtype=torch.uint8)
b = torch.tensor([100], dtype=torch.uint8)

print((a + b).item(), (a + b).dtype)

44 torch.uint8

Three hundred does not fit in an unsigned byte, so it wrapped. Image data loaded as uint8 and manipulated arithmetically before conversion to float is a realistic route to this, and nothing in the output announces it.

Promotion between dtypes follows rules that are usually what you want and occasionally expensive:

print((torch.ones(3) + torch.ones(3, dtype=torch.float64)).dtype)
print((torch.ones(3) + torch.ones(3, dtype=torch.long)).dtype)
print((torch.ones(3, dtype=torch.float16) + torch.ones(3)).dtype)

torch.float64
torch.float32
torch.float32

The first line is worth remembering. A NumPy array created from ordinary floating-point literals is typically float64, and torch.from_numpy() preserves that dtype.

What happens when that tensor enters a float32 program depends on the operation. Some arithmetic operations apply PyTorch’s type-promotion rules and produce a higher-precision result; other operations require compatible dtypes and may raise instead. Either way, an unexpected float64 tensor is worth finding at the boundary where it entered the pipeline rather than discovering it several operations later.

import numpy as np

print(torch.from_numpy(np.array([1.0, 2.0, 3.0])).dtype)

torch.float64

Reduced precision has the opposite failure mode, where the dtype is small enough that ordinary values leave its range:

print(torch.full((4096,), 1.0, dtype=torch.float16).sum().item())
print(torch.full((65536,), 1.0, dtype=torch.float16).sum().item())
print(torch.finfo(torch.float16).max)

4096.0
inf
65504.0

Summing ones overflowed, because float16 cannot represent 65536. When mixed precision appears later in the book, this is the mechanism behind the machinery that exists to prevent it.

Some operations impose specific dtype contracts. A common example is classification with cross_entropy: when the target represents class indices, those indices must use an integer dtype:

logits = torch.randn(4, 10)
torch.nn.functional.cross_entropy(logits, torch.tensor([1.0, 5.0, 3.0, 0.0]))

RuntimeError: expected target dtype to be Long or Byte, but got Float

So “convert everything to float” is not a safe rule. In this example the target represents class indices, so integer dtype is part of the operation’s contract. Other losses use floating-point targets instead. dtype, like shape, can encode what a tensor means to the operation consuming it.

Device follows the same pattern with a narrower rule: operands must generally live in the same place.

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

x = torch.randn(32, 128, device=device)
w = torch.randn(128, 64, device=device)

out = x @ w

When they do not, the message is explicit:

RuntimeError: Expected all tensors to be on the same device,
but found at least two devices, cuda:0 and cpu!

The interesting version of this failure is not the one where you forgot to move your inputs. It is the tensor created inside a function without reference to the tensors already flowing through that function.

Factory functions such as torch.arange, torch.ones and torch.eye use PyTorch’s configured default device unless you specify one; they do not automatically inherit x.device merely because x is nearby in the computation. Since the default device is initially CPU, this commonly appears when the surrounding model has been moved to an accelerator. The habit that avoids it is to derive the device from a tensor already in hand:

positions = torch.arange(x.shape[1], device=x.device)

Chapter 5 covers how model.to(device) moves registered parameters and buffers, and why tensors created in a forward pass are not covered by it.

Writing the contract down

Shape assertions are worth writing for the same reason as any other precondition, and the argument against them, that they clutter the code, is answered by putting them at boundaries rather than everywhere. A function that other code calls with tensors it did not create is a boundary. A line in the middle of a well-understood block is not.

The essential property of a good shape check is that its message contains the evidence:

def mse(predictions, targets):
    if predictions.shape != targets.shape:
        raise ValueError(
            f"predictions {tuple(predictions.shape)} and "
            f"targets {tuple(targets.shape)} must match"
        )
    return ((predictions - targets) ** 2).mean()

That turns the silent broadcasting bug from the start of the chapter into an immediate failure at the point where the assumption was violated, and it prints both shapes so you do not have to add a print to find out what happened.

For tensors where only some axes are fixed, a helper avoids repeating the same four lines:

def require_shape(name, t, *expected):
    """Each entry is an int, or None for 'any size'."""
    if t.ndim != len(expected):
        raise ValueError(
            f"{name}: expected {len(expected)} dims, got {t.ndim} "
            f"with shape {tuple(t.shape)}"
        )
    for axis, (actual, want) in enumerate(zip(t.shape, expected)):
        if want is not None and actual != want:
            raise ValueError(
                f"{name}: axis {axis} expected {want}, got {actual}; "
                f"full shape {tuple(t.shape)}"
            )

require_shape("hidden", torch.randn(32, 100, 768), None, None, 768)

This passes for any batch and sequence length while pinning the embedding width, which is usually the axis you actually care about.

The lighter-weight alternative, and the one you will see most often in real model code, is a comment that records the shape at each step:

# x: (B, T, D)
q = x.reshape(B, T, H, Dh)
# q: (B, T, H, Dh)
q = q.transpose(1, 2)
# q: (B, H, T, Dh)

Those comments have no runtime effect and are not a substitute for a check at a boundary. They earn their place because tensor code compresses several axis manipulations into a line or two, and the comments record the reasoning that would otherwise have to be reconstructed. They are the closest thing the program has to a written statement of its own type system, and the exercise of writing them is often what surfaces the bug.

One caution, from the pipeline earlier: a shape check compares integers, so it cannot detect an error that swaps two axes of equal size. Keep the test sizes distinct.

Using AI on tensor code

The failures in this chapter are exactly the ones an assistant can easily paper over, because many shape exceptions can be made to disappear with a reshape, a squeeze, a transpose, or some other local adjustment. Making the operation legal is much easier than proving that the resulting axes mean what the surrounding model expects. Whether it restores the intended meaning is a separate question, and it is not one the error message can answer.

So do not open with the error. Open with a request for a trace that commits to claims you can check. Try this against the broken forward pass from earlier:

Here is a PyTorch function and the shapes of the tensors it closes over.
Do not rewrite it and do not propose a fix yet.

Walk through it operation by operation. For each intermediate tensor, give
the symbolic shape you predict, using B for batch and named symbols for the
other axes, and say which rule produced it: broadcasting, matmul contraction,
reduction, axis insertion or removal, reinterpretation, or reordering.

Then tell me, for each axis of each intermediate, what it means in terms of
the model: which one indexes examples, which indexes features, and so on.

Finally, identify the first operation where the shape you predicted disagrees
with the contract implied by the surrounding code, and say what evidence in
the code makes you confident that this is a disagreement rather than a
convention I have chosen deliberately.

The value is in the shape of the answer. A symbolic trace makes a prediction for every batch size, not only the one you ran, which is what would have exposed the (hidden, B) transposition even while it was hiding behind a square tensor. Naming the axes forces the assistant to state the interpretation it is assuming, and that is where you will usually spot the disagreement, because its assumption and yours are visible side by side. Asking what evidence supports the claim distinguishes a reasoned answer from a plausible one.

Then check it. Run the code, print the shapes, and compare them against the predicted trace. Where they agree you have gained confidence; where they disagree you have found the boundary to investigate. Either way you now know something specific enough to ask a specific follow-up question, which is a much better use of an assistant than “why doesn’t this work”.

Ask for a fix once you know what is broken. The rule from Chapter 1 has not changed: a proposed fix you cannot check is a guess with better grammar than yours.

When a tensor surprises you

The questions below are the ones this chapter has been building the equipment to answer. They are roughly in the order that resolves problems fastest, which is: establish the facts, then the intent, then the mechanism.

What did I expect, and what did I get? Print tuple(t.shape) and compare it against the shape you would have written down. If you cannot write down the expected shape, that is the first problem to solve, not the code.

What does each axis mean? Assign a symbol to every axis of every operand. Most shape bugs become obvious at this step, before anything is run.

Is the element count what I expected? numel() catches accidental broadcasting when the shape alone does not look alarming. Thirty-two examples should not produce a thousand values.

Which operation changed the shape, and was the change explicit? Elementwise unary operations cannot change a shape. Broadcasting, reductions, indexing and reinterpretation can, and only some of those were things you asked for by name.

Are two axes the same size? If so, no shape check can tell them apart. Re-run with distinct sizes before trusting anything.

Did an axis silently disappear? Integer indexing, squeeze() with no argument, reductions without keepdim, and a 1-D operand to matmul all remove axes. A missing axis is often what makes a later broadcast do something unintended.

Where did expected and actual first diverge? Trace forward from the inputs rather than backwards from the traceback. The failing line is the first illegal operation, which is not necessarily the first wrong one.

Is this a shape problem or a layout problem? If a view() error mentions size and stride, first confirm that the requested element count is valid. If it is, the immediate failure is usually that the current strides cannot represent the requested view without copying. Check stride() and is_contiguous(), and identify the operation that produced the layout before reaching automatically for .contiguous().

Could two different operations reach this shape? If a reshape and a permutation both produce the target, decide which one you meant by running the small arange experiment and reading off where the values went.

Is dtype or device part of it? Integer where float was expected, float64 from NumPy, overflow in a reduced-precision type, or a constant created on CPU inside a function running on an accelerator.

Does the repair restore the contract, or just remove the exception? Name the quantity that was wrong, say what it should be, and observe that it now is. For the broadcasting bug at the start of this chapter, that meant w reaching 3.0, not the loss going down.

What you should now be able to answer

Here is a fragment that appears, in one form or another, in every transformer implementation you will read. It should no longer be opaque.

B, T, D = 32, 128, 768
H = 12
Dh = D // H

x = torch.randn(B, T, D)

q = x.reshape(B, T, H, Dh)
q = q.transpose(1, 2)

scores = q @ q.transpose(-2, -1)

What is x? A batch of 32 sequences, each 128 positions long, each position represented by 768 numbers. (B, T, D).

What does the reshape do? It divides the 768-wide embedding into 12 groups of 64, giving (B, T, H, Dh). It is a reinterpretation: no value moved, and 32 × 128 × 12 × 64 equals 32 × 128 × 768.

Why that reshape and not reshape(B, H, T, Dh)? Because reshape follows the flat ordering of the elements, and in (B, T, D) the embedding varies fastest. Splitting the last axis gives each head its own slice of the embedding. The other spelling would give each head a slice of the sequence.

What does the transpose do? It exchanges axes 1 and 2, producing (B, H, T, Dh), so that the head axis sits alongside the batch axis and the matrix multiplication that follows operates on (T, Dh) matrices. It moves no data; it changes the strides.

Is q contiguous afterwards? No. It shares storage with x, and the transposed strides do not match the standard layout for its shape. A subsequent view() may refuse; reshape() will succeed, copying if necessary.

What shape is scores? q is (B, H, T, Dh), and q.transpose(-2, -1) is (B, H, Dh, T). Matmul contracts the inner Dh and broadcasts the leading (B, H), giving (B, H, T, T), which is (32, 12, 128, 128). Every position scored against every position, for each head, for each example.

How would I know if H and Dh had been swapped? Not from the shape, if they were equal, which is why choosing unequal test sizes matters. With H = 12 and Dh = 64 they are not equal, so reshape(B, T, Dh, H) would produce (32, 128, 64, 12) and the subsequent transpose would give (32, 64, 128, 12), which the matmul would still accept, producing (32, 64, 128, 128). Legal, wrong, and detectable by checking the second axis against H.

What would I assert? After the transpose: q.shape == (B, H, T, Dh). After the matmul: scores.shape == (B, H, T, T). Both at a boundary, both naming axes that carry meaning.

You should be able to derive every one of those without running the code, which is the capability the chapter was for. It matters considerably more than knowing how many tensor methods exist.

Exercises

These are written to be run, and they map onto the notebook that accompanies this chapter.

1. Predict before you execute. For each of the following, write the output shape down before running it: torch.randn(8, 1, 6) * torch.randn(4, 6); torch.randn(5, 3) @ torch.randn(3); torch.randn(4, 6, 8).sum(dim=1, keepdim=True); torch.randn(1, 10, 1).squeeze(); torch.randn(2, 3, 4).permute(2, 0, 1). For every one you get wrong, identify which rule you misapplied.

2. Reproduce the opening bug and predict its fixed point. Run the loop from the start of this chapter with targets [2, 4, 6, 8] shaped (4, 1). Before running, predict the converged value of b and the plateau loss from the mean and variance of the targets. Then repair it and confirm w reaches 2.0.

3. Make an accidental broadcast visible. Construct a pair of shapes that broadcast to something larger than either operand, and write a one-line check using numel() that would have caught it. Then find a pair that raises instead, and explain what distinguishes the two cases.

4. Locate the first divergence. Take the broken forward from this chapter and add the check instrumentation. Run it at batch sizes 32, 16 and 33. Explain why one of them reports nothing wrong, and what that tells you about relying on shape assertions alone.

5. Map the storage. For x = torch.arange(24).reshape(2, 3, 4), print stride() and is_contiguous() for x, x.transpose(0, 1), x.permute(2, 0, 1), and x[:, :, ::2]. For each, predict whether view(-1) will succeed before trying it, and explain the result in terms of strides rather than contiguity.

6. Prove that reshape sometimes copies. Construct one case where reshape shares the original storage and one where it must allocate new storage. Inspect the storage relationship, then modify the reshaped result and observe whether the original changes. Explain why production code should not depend on whether reshape() happened to return a view or a copy.

7. Split the heads twice. Using torch.arange and a tensor small enough to print, implement both the correct and the incorrect head split from this chapter. Write a single assertion that passes for the correct one and fails for the incorrect one, using a property that is not the shape.

8. Two normalizations. Standardize a (32, 64) tensor across the batch and across the features. Confirm both have the same shape, then find a measurement that distinguishes them. Repeat on a (32, 32) tensor and explain what changes.

9. Watch dtype promote. Build a small pipeline that starts from a NumPy array and ends in a matrix multiplication, and find where the tensor becomes float64. Fix it at the source rather than casting at the end, and say why the two repairs are not equivalent.

Next: the record that made all of this differentiable

We now have the objects. A tensor has a shape which is a contract nobody enforces, a dtype and a device which decide whether an operation is legal at all, and a set of strides that determine which reinterpretations are free and which require a copy. Given an operation and its operands, you can derive what comes out, and when the derivation disagrees with reality you have a procedure for finding the first place they parted company.

What we have not looked at is the other thing PyTorch was doing while all of this executed. In Chapter 1, loss.backward() walked a record of the operations that produced loss, and we took the existence of that record on trust. When tensors participating in autograd flowed through these operations, PyTorch also recorded the differentiable relationships needed for the backward pass: broadcasts, reductions, reshapes, transposes and the arithmetic around them. Broadcasting in particular has a backward pass that has to sum a gradient back down to the shape of the operand it was expanded from, which is a fact you can eventually use as evidence about which tensor was expanded.

The next chapter opens that record. What is in it, how requires_grad, grad_fn and leaf status determine its structure, where it can be cut without anyone telling you, and what to inspect when gradients are missing, stale, or NaN.

Autograd.