PyTorch: Zero to Hero — Advanced Interlude 06A

There is a point in machine learning where the numbers become difficult for a human being to picture.

One dimension is easy.

Two dimensions are easy.

Three dimensions are still possible to draw.

Then somebody hands you a tensor containing 100 features.

Or 768.

Or 4,096.

And suddenly the language changes.

People start talking about embedding spaces, feature directions, hyperplanes, high-dimensional geometry and latent representations.

It can sound as though the mathematics has become fundamentally different.

It has not.

The computer has one advantage over us here:

PyTorch does not need to visualise a space in order to calculate inside it.

A point in two dimensions contains two coordinates.

A point in three dimensions contains three coordinates.

A point in one hundred dimensions contains one hundred coordinates.

The arithmetic keeps working after our imagination gives up.

This chapter is about learning to trust that fact.

We will move from one dimension to two, three, ten, one hundred and finally hundreds of dimensions. Then we will build a linear support vector machine in PyTorch and use it to classify points in a space we cannot draw.

The SVM is useful here because it makes the geometry unusually clear.

A line in two dimensions becomes a plane in three dimensions and a hyperplane in higher dimensions.

The name changes.

The operation barely does.

1. Start with one number

Consider:

import torch

x = torch.tensor([3.0])

There is one coordinate.

Geometrically, we can think of it as one point on a number line:

0 ---- 1 ---- 2 ---- 3 ---- 4 ---- 5
                     ●

The point is described by:

(3)

Nothing mysterious so far.

Now give the point another coordinate.

x = torch.tensor([3.0, 2.0])

Now it lives in a two-dimensional space.

We write:

(3, 2)

and we can draw it on ordinary graph paper.

x₂
^
|
|        ● (3,2)
|
|
+-----------------> x₁

The tensor contains two numbers because the point needs two coordinates.

2. Three dimensions do not change the idea

Add another coordinate:

x = torch.tensor([3.0, 2.0, 5.0])

Now the point is:

(3, 2, 5)

We can call the axes:

x₁
x₂
x₃

or:

width
height
depth

or:

temperature
pressure
humidity

The mathematics does not care what the coordinates mean.

The important fact is simply:

number of features = 3

and therefore one example is represented by three values.

3. Now add a fourth coordinate

x = torch.tensor([3.0, 2.0, 5.0, 1.5])

We have reached a strange human boundary.

The tensor is perfectly ordinary.

But drawing four perpendicular spatial axes is not.

That can make four-dimensional space feel more exotic than it really is.

From PyTorch’s point of view, however, this:

x = torch.tensor([3.0, 2.0, 5.0])

and this:

x = torch.tensor([3.0, 2.0, 5.0, 1.5])

are the same kind of operation.

One vector happens to contain three coordinates.

The other contains four.

There is no cliff at dimension four.

Only our visual intuition stops helping.

4. One hundred dimensions are still just coordinates

Now do this:

x = torch.randn(100)

print(x.shape)

Output:

torch.Size([100])

That tensor can represent one point in a 100-dimensional feature space.

Conceptually:

x = (x₁, x₂, x₃, ..., x₁₀₀)

We cannot draw all one hundred axes.

PyTorch does not need to.

It can still add vectors:

y = torch.randn(100)
z = x + y

measure distance:

distance = torch.linalg.vector_norm(x - y)

calculate a dot product:

similarity = x @ y

or multiply the point by a matrix:

W = torch.randn(50, 100)
out = W @ x

Every operation is well defined.

The fact that we cannot sketch the space on paper is irrelevant to the computation.

5. A very important PyTorch distinction: tensor rank is not feature-space dimensionality

This is where the word dimension becomes dangerous.

Look at this tensor:

x = torch.randn(100)

PyTorch reports:

print(x.ndim)   # 1

So it is a one-dimensional tensor.

But if those 100 values describe one example, that example can be thought of as a point in a 100-dimensional feature space.

Those statements do not conflict.

They answer different questions.

x.shape = [100]

Tensor rank:
    1 axis

Feature-space dimensionality:
    100 coordinates per point

Now batch 32 such points:

X = torch.randn(32, 100)

print(X.ndim)   # 2
print(X.shape)  # torch.Size([32, 100])

This is a rank-2 tensor containing:

32 examples
100 features per example

Geometrically:

32 points in a 100-dimensional feature space.

That distinction becomes extremely important later in the book.

A transformer tensor such as:

[B, T, 768]

is a rank-3 tensor.

But each token position can be represented by a vector containing 768 coordinates.

So one useful reading is:

B     batches
T     token positions
768   coordinates describing each token representation

Do not confuse the number of axes in the storage object with the dimensionality of the representation stored along one of those axes.

6. What does a direction mean in many dimensions?

In two dimensions, a vector can represent a direction:

v = torch.tensor([2.0, 1.0])

In 100 dimensions, exactly the same idea applies:

v = torch.randn(100)

The direction now has one component along each of the 100 axes.

We cannot point our finger in that direction in physical space.

But we can still project another vector onto it using a dot product.

x = torch.randn(100)
v = torch.randn(100)

projection_score = x @ v

Expanded, that is:

x₁v₁ + x₂v₂ + x₃v₃ + ... + x₁₀₀v₁₀₀

This is one of the most important operations in machine learning.

A high-dimensional operation often looks sophisticated because there are many coordinates.

But the mechanism can still be ordinary arithmetic repeated across them.

7. A boundary in one dimension

Suppose we have points on a line.

class -1             class +1

● ● ● ●       |       ○ ○ ○ ○
--------------+-----------------
              0

A classifier can place a boundary at zero.

For a scalar input x, one possible score is:

score = wx + b

If:

score > 0

predict one class.

If:

score < 0

predict the other.

This tiny equation is going to survive all the way to 100 dimensions.

8. The boundary becomes a line in two dimensions

Now each example has two coordinates:

x = (x₁, x₂)

A linear classifier can calculate:

score = w₁x₁ + w₂x₂ + b

The decision boundary occurs where:

w₁x₁ + w₂x₂ + b = 0

In two dimensions that equation describes a line.

Imagine two classes:

x₂
^
|
|   ○ ○
|  ○ ○
|       ---------------- boundary
|                    × ×
|                      × ×
+--------------------------------> x₁

Everything on one side produces a positive score.

Everything on the other side produces a negative score.

In PyTorch:

x = torch.tensor([2.0, -1.0])
w = torch.tensor([0.8, 1.3])
b = torch.tensor(-0.2)

score = x @ w + b

The dot product is just a compact way of writing:

2.0 × 0.8 + (-1.0) × 1.3

9. In three dimensions the boundary becomes a plane

Give every point a third coordinate:

x = (x₁, x₂, x₃)

The classifier becomes:

score = w₁x₁ + w₂x₂ + w₃x₃ + b

The boundary is:

w₁x₁ + w₂x₂ + w₃x₃ + b = 0

In three dimensions, that describes a plane.

So far:

1 feature  → boundary is a point
2 features → boundary is a line
3 features → boundary is a plane

What happens at 100 dimensions?

We do not invent new arithmetic.

We continue the same equation:

w₁x₁ + w₂x₂ + ... + w₁₀₀x₁₀₀ + b = 0

We call the resulting boundary a hyperplane.

The word is intimidating.

The equation is not.

10. A 100-dimensional classifier fits in one line

features = 100

x = torch.randn(features)
w = torch.randn(features)
b = torch.tensor(0.0)

score = x @ w + b

That is a linear classifier operating in a 100-dimensional feature space.

The computer does not construct a 100-dimensional picture.

It calculates a dot product.

This is the key mental shift:

High-dimensional machine learning does not require high-dimensional imagination. It requires operations that remain valid as the number of coordinates grows.

11. What makes an SVM different?

A linear classifier only needs a boundary that separates the classes.

A support vector machine asks for something more specific.

It wants a boundary with a large margin.

In two dimensions, imagine several possible separating lines.

Some scrape close to the data.

One leaves a wider gap between the nearest examples of the two classes.

class -1                           class +1

 ●   ●
   ●

------------- margin ----------------
============= boundary ==============
------------- margin ----------------

                              ○
                           ○      ○

The points nearest the margin are the support vectors.

They matter because they constrain where the maximum-margin boundary can sit.

In higher dimensions the picture disappears, but the idea remains:

find a hyperplane
that separates the classes
while maintaining a large margin

12. The SVM score is still a dot product

For a linear SVM:

f(x) = w · x + b

In PyTorch:

score = x @ w + b

For a batch:

X = torch.randn(32, 100)
w = torch.randn(100)
b = torch.tensor(0.0)

scores = X @ w + b

print(scores.shape)

Output:

torch.Size([32])

We supplied 32 points in a 100-dimensional feature space and received one score for each point.

The same matrix-vector multiplication handles all 32 examples.

13. Labels for a binary SVM are usually -1 and +1

Let:

y = +1

mean the positive class and:

y = -1

mean the negative class.

A correct confident prediction should make:

y × score

positive and preferably at least 1.

If:

y = +1
score = +2.4

then:

y × score = 2.4

Good.

If:

y = -1
score = -1.8

then:

y × score = 1.8

Also good.

If the signs disagree, the product becomes negative.

That gives us a compact way to express both correctness and margin.

14. Hinge loss

A classic SVM uses hinge loss:

max(0, 1 - y f(x))

In PyTorch:

hinge = torch.relu(1 - y * scores)
loss = hinge.mean()

Examples that are correctly classified and lie beyond the margin have:

y f(x) >= 1

so their hinge loss is zero.

Examples inside the margin or on the wrong side of the boundary contribute positive loss.

This is a useful contrast with losses such as cross entropy.

The SVM does not keep rewarding a point forever for moving farther and farther away once it is safely outside the margin.

15. The other half of the SVM objective: keep the boundary simple

If we only minimize hinge loss, many weight vectors may separate the training set.

The maximum-margin formulation also penalizes the size of w.

A common primal objective is conceptually:

0.5 ||w||² + C × mean(hinge_loss)

where C controls the trade-off between:

small weights / larger margin

and:

penalizing margin violations

In PyTorch:

regularizer = 0.5 * model.w.square().sum()
hinge = torch.relu(1 - y * scores).mean()

loss = regularizer + C * hinge

There are several equivalent conventions for scaling the regularization and hinge terms.

What matters here is the idea:

The SVM is not merely trying to classify correctly. It is choosing a separating direction with a margin.

16. Build a linear SVM from scratch with nn.Module

Now the entire model:

import torch
import torch.nn as nn


class LinearSVM(nn.Module):
    def __init__(self, features):
        super().__init__()

        self.w = nn.Parameter(
            torch.zeros(features)
        )
        self.b = nn.Parameter(
            torch.zeros(())
        )

    def forward(self, x):
        return x @ self.w + self.b

For two-dimensional data:

model = LinearSVM(2)

For 100-dimensional data:

model = LinearSVM(100)

For 768-dimensional embeddings:

model = LinearSVM(768)

The class did not change.

Only this number changed:

features

That is the point of the chapter.

17. Train it in two dimensions first

We should prove the model somewhere we can still see the geometry.

Create two clouds of points:

import torch


torch.manual_seed(7)

negative = torch.randn(200, 2) * 0.7 + torch.tensor([-2.0, -1.5])
positive = torch.randn(200, 2) * 0.7 + torch.tensor([ 2.0,  1.5])

X = torch.cat([negative, positive], dim=0)
y = torch.cat([
    -torch.ones(len(negative)),
     torch.ones(len(positive)),
])

Check the contract:

print(X.shape)  # [400, 2]
print(y.shape)  # [400]

That means:

400 points
2 coordinates per point

Train:

model = LinearSVM(features=2)
optimizer = torch.optim.SGD(model.parameters(), lr=0.02)
C = 1.0

for step in range(1000):
    scores = model(X)

    hinge = torch.relu(1 - y * scores).mean()
    regularizer = 0.5 * model.w.square().sum()
    loss = regularizer + C * hinge

    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

    if step % 200 == 0:
        with torch.no_grad():
            predictions = torch.sign(scores)
            accuracy = (predictions == y).float().mean()

        print(
            f"step={step:04d} "
            f"loss={loss.item():.4f} "
            f"accuracy={accuracy.item():.3f}"
        )

We have used exactly the machinery from earlier chapters:

parameters
forward pass
loss
backward
optimizer step

The geometry is new.

The training mechanism is not.

18. Draw the two-dimensional boundary

Because this case has only two features, we can inspect the learned line.

The boundary satisfies:

w₁x₁ + w₂x₂ + b = 0

Solve for x₂:

x₂ = -(w₁x₁ + b) / w₂

Plot it:

import matplotlib.pyplot as plt

with torch.no_grad():
    w = model.w.detach().cpu()
    b = model.b.detach().cpu()

plt.scatter(
    X[y < 0, 0],
    X[y < 0, 1],
    marker='x',
    label='class -1',
)
plt.scatter(
    X[y > 0, 0],
    X[y > 0, 1],
    marker='o',
    label='class +1',
)

x1 = torch.linspace(X[:, 0].min(), X[:, 0].max(), 200)
x2 = -(w[0] * x1 + b) / w[1]

plt.plot(x1, x2, label='decision boundary')
plt.xlabel('x1')
plt.ylabel('x2')
plt.legend()
plt.title('Linear SVM in two dimensions')
plt.grid(True)
plt.show()

Now we have a picture of what the model learned.

The next experiment removes that privilege.

19. Move to 100 dimensions

Create 2,000 points:

torch.manual_seed(11)

X = torch.randn(2000, 100)

Each row is one point:

[x₁, x₂, x₃, ..., x₁₀₀]

Now create a classification rule that depends mostly on three coordinates:

true_score = (
    2.0 * X[:, 7]
    - 1.5 * X[:, 42]
    + 0.8 * X[:, 81]
    + 0.15 * torch.randn(2000)
)

y = torch.where(
    true_score >= 0,
    torch.tensor(1.0),
    torch.tensor(-1.0),
)

The dataset has:

2,000 examples
100 features per example

We cannot draw the point cloud.

But every operation we need remains available.

20. Train exactly the same model

model = LinearSVM(features=100)
optimizer = torch.optim.Adam(model.parameters(), lr=0.03)
C = 2.0

for step in range(1200):
    scores = model(X)

    hinge = torch.relu(1 - y * scores).mean()
    regularizer = 0.5 * model.w.square().sum()
    loss = regularizer + C * hinge

    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

    if step % 200 == 0:
        with torch.no_grad():
            predictions = torch.where(
                scores >= 0,
                1.0,
                -1.0,
            )
            accuracy = (predictions == y).float().mean()

        print(
            f"step={step:04d} "
            f"loss={loss.item():.4f} "
            f"accuracy={accuracy.item():.3f}"
        )

Notice what did not happen.

We did not add special 100-dimensional code.

We did not invent a new optimizer.

We did not need to visualise the data.

We changed:

LinearSVM(2)

to:

LinearSVM(100)

and the same algebra continued to work.

21. Inspect the learned direction

The vector:

model.w

contains one learned weight for every input dimension.

So we can ask which dimensions contributed most strongly to the separating direction.

with torch.no_grad():
    importance = model.w.abs()
    values, indices = torch.topk(importance, k=10)

for index, value in zip(indices.tolist(), values.tolist()):
    print(index, value)

Because we constructed the synthetic labels using dimensions 7, 42 and 81, those dimensions should be important if training successfully recovers the underlying separating direction.

Do not interpret every learned weight as a perfect measure of causal importance.

Correlated features, noise, regularization and finite data can all affect the result.

The useful point is simpler:

The classifier has learned one direction through a 100-dimensional space.

That direction is encoded by w.

22. The weight vector is perpendicular to the hyperplane

This is a beautiful geometric fact.

The decision boundary is:

w · x + b = 0

The vector w points perpendicular to that boundary.

In two dimensions, if the line tilts, w tells us the direction normal to the line.

In three dimensions, w is normal to the separating plane.

In one hundred dimensions, w is normal to the separating hyperplane.

We cannot draw it.

The relationship is unchanged.

This is why inspecting the learned weight vector is meaningful: it describes the orientation of the boundary in feature space.

23. Distance from the boundary

The raw score:

w · x + b

changes if we scale w and b.

The geometric signed distance to the hyperplane is:

(w · x + b) / ||w||

In PyTorch:

scores = model(X)
distances = scores / torch.linalg.vector_norm(model.w)

Large positive distance means one side of the boundary.

Large negative distance means the other.

Values near zero lie close to the decision boundary.

For an SVM, examples near the margin are the interesting ones.

24. Find examples close to the margin

For the canonical margin surfaces:

w · x + b = +1
w · x + b = -1

points with:

y × score ≈ 1

are close to the margin.

Inspect them:

with torch.no_grad():
    scores = model(X)
    margin_value = y * scores

    distance_from_margin = (margin_value - 1).abs()
    closest = torch.topk(
        distance_from_margin,
        k=10,
        largest=False,
    ).indices

print(closest)
print(margin_value[closest])

These are the examples most tightly constraining the learned boundary.

Again, we do not need to draw 100 dimensions to ask a geometric question about them.

25. Now make the feature space 768-dimensional

Modern AI systems frequently represent objects using vectors with hundreds or thousands of components.

For example, suppose some upstream model gives us one 768-value representation per piece of text:

embeddings = torch.randn(5000, 768)

The tensor is rank 2:

[5000, 768]

A useful geometric reading is:

5,000 points
in a 768-dimensional representation space

Our classifier does not care whether those 768 numbers originated from hand-engineered measurements or a neural embedding model.

classifier = LinearSVM(768)
scores = classifier(embeddings)

Output:

[5000]

One score per embedded item.

26. Do not assume one embedding dimension means one human concept

There is an important warning here.

It is tempting to imagine an embedding like:

dimension 0 = clarity
dimension 1 = humour
dimension 2 = technicality
...

Real learned representations are usually not so obliging.

Meaning can be distributed across many coordinates.

A useful property may correspond more closely to:

a direction

or:

a region

or:

a subspace

rather than one named axis.

A linear classifier learns one weighted direction across all coordinates:

w₁x₁ + w₂x₂ + ... + w₇₆₈x₇₆₈

That is part of why linear probes and simple ranking models can reveal useful structure inside large representations.

27. Why a linear model can still be useful in a complicated representation space

Imagine the original raw input is text.

Text itself is not naturally a point in a neat linearly separable space.

But an embedding model can transform it:

raw text
   ↓
embedding model
   ↓
768-dimensional vector

Then a simple classifier may operate on that representation:

768-dimensional vector
   ↓
linear boundary
   ↓
class

The linear model is simple.

The representation may already contain a great deal of useful structure.

This is an important machine-learning pattern:

A simple decision rule can become powerful when the representation feeding it is good.

28. One dimension, two, three, one hundred: compare the code

One feature:

model = LinearSVM(1)
x = torch.randn(32, 1)
scores = model(x)

Two features:

model = LinearSVM(2)
x = torch.randn(32, 2)
scores = model(x)

Three features:

model = LinearSVM(3)
x = torch.randn(32, 3)
scores = model(x)

One hundred features:

model = LinearSVM(100)
x = torch.randn(32, 100)
scores = model(x)

Seven hundred and sixty-eight features:

model = LinearSVM(768)
x = torch.randn(32, 768)
scores = model(x)

The conceptual operation is unchanged:

x @ w + b

Only the length of the vectors changes.

29. Broadcasting is helping us quietly

Remember:

X.shape
# [32, 100]

model.w.shape
# [100]

model.b.shape
# []

Then:

X @ model.w

produces:

[32]

and:

X @ model.w + model.b

still produces:

[32]

The scalar bias broadcasts across all examples.

So this chapter also reconnects to the shape rules from the beginning of the book.

High-dimensional models do not escape tensor shape reasoning.

They depend on it.

30. A tensor can contain several different kinds of dimensions at once

This is where high-dimensional thinking becomes useful for attention.

Suppose:

x = torch.randn(8, 128, 768)

The shape is:

[8, 128, 768]

Do not flatten those numbers into the vague statement:

“This is a three-dimensional tensor.”

That is true but incomplete.

Give every axis a meaning:

8     = batch
128   = sequence positions
768   = features describing each position

At a particular batch and token position:

v = x[0, 17]

print(v.shape)

Output:

[768]

That one vector can be thought of as a point in a 768-dimensional representation space.

The whole tensor stores many such points arranged by batch and sequence position.

That is the bridge into attention.

31. High-dimensional does not mean infinitely expressive

A 100-dimensional linear classifier is still linear.

If the classes require a curved or disconnected decision boundary, a single hyperplane may fail.

For example, XOR is not linearly separable in its ordinary two-dimensional representation.

Adding dimensions does not automatically solve every learning problem.

The important distinction is:

number of dimensions

versus:

shape of the decision boundary

A linear SVM learns one hyperplane regardless of whether there are 2 features or 2,000.

Neural networks gain power partly by transforming representations so that useful decisions become easier in later feature spaces.

32. The kernel idea, briefly

Classical SVMs are also famous for kernels.

A kernel allows an SVM to behave as though data had been mapped into another feature space without always constructing that feature vector explicitly.

That is a fascinating subject, but it is not the point of this chapter.

Our goal is the opposite: we want to make the high-dimensional representation completely explicit so we can see how naturally PyTorch operates on it.

The linear SVM gives us the cleanest demonstration.

33. What becomes expensive as dimensionality grows?

The idea stays simple, but computation is not free.

A batch of:

[32, 100]

contains:

3,200 values

A batch of:

[32, 10_000]

contains:

320,000 values

The dot product grows with the number of features.

Memory grows too.

But this is a scaling issue rather than a conceptual discontinuity.

PyTorch is designed precisely to perform these large tensor operations efficiently.

34. Debug high-dimensional code by naming the axes

When a tensor becomes:

[64, 512, 768]

human intuition often fails because the programmer stops assigning meaning to each axis.

Do not do that.

Write a shape ledger:

[B, T, D]

B = batch
T = positions
D = feature-space coordinates

Then annotate operations:

[B, T, D]
    @
[D, H]
    ↓
[B, T, H]

The number 768 is not the difficult part.

Losing track of what the axis means is the difficult part.

35. Challenge: recover a hidden direction in 500 dimensions

Create data:

torch.manual_seed(23)

X = torch.randn(4000, 500)

hidden_w = torch.zeros(500)
hidden_w[12] = 1.8
hidden_w[117] = -2.2
hidden_w[308] = 0.9
hidden_w[444] = 1.4

scores = X @ hidden_w + 0.2 * torch.randn(4000)
y = torch.where(scores >= 0, 1.0, -1.0)

Now train:

model = LinearSVM(500)

without looking at hidden_w again.

After training, inspect:

values, indices = torch.topk(
    model.w.detach().abs(),
    k=10,
)

print(indices)
print(values)

Questions to answer:

Which dimensions received the largest weights?
Do 12, 117, 308 and 444 appear near the top?
What happens when you increase the noise?
What happens when you reduce the number of training examples?
What happens when you increase C?
What happens when you remove regularization?

Do not try to visualise 500 dimensions.

Interrogate the learned structure directly.

36. The deeper connection to the previous chapter

Earlier we saw another important PyTorch idea:

large model
=
simple modules composed repeatedly

This chapter gives us a parallel idea:

high-dimensional representation
=
many ordinary coordinates considered together

In both cases, scale can trick us into believing the underlying mechanism has changed.

It has not.

A transformer may contain many nested modules.

An embedding may contain hundreds or thousands of coordinates.

PyTorch gives us operations that keep working as both structures grow.

That is where much of its power comes from.

37. The one concept to keep

Humans live in a world where three spatial dimensions dominate our intuition.

Machine learning does not.

Once a point is represented as a tensor, adding another feature simply adds another coordinate.

The progression is continuous:

1 feature
2 features
3 features
4 features
10 features
100 features
768 features
4,096 features

We lose the ability to draw the space somewhere near the beginning of that list.

The algebra keeps going.

A linear SVM demonstrates this beautifully:

1D     → separating point
2D     → separating line
3D     → separating plane
100D   → separating hyperplane
768D   → separating hyperplane

and every version can still be written as:

score = x @ w + b

That is the lesson.

Do not mistake the limits of human visualisation for limits in the mathematics.

PyTorch is perfectly comfortable operating in spaces we cannot picture.

Our job is not to imagine every axis.

Our job is to keep track of what the axes mean and use operations whose meaning survives as the dimensionality grows.

In the next chapter, attention will give us tensors shaped like:

[batch, sequence, embedding]

with embedding dimensions measured in the hundreds or thousands.

After this chapter, that number should no longer be intimidating.

It is just another axis full of coordinates.