PyTorch From First Principles — Interlude 05A

In the previous chapter we treated the DataLoader as a production system.

It reads samples, prepares batches, moves work through worker processes and tries to keep the accelerator fed.

But there is a question hiding inside that pipeline:

What exactly is a sample by the time the model receives it?

A file on disk is not usually the tensor the model trains on.

An image may begin as compressed bytes.

A sentence may begin as Unicode text.

An audio example may begin as a waveform stored in a file.

A row of tabular data may begin as strings, missing values and categorical fields.

Before any of those reach the model, something has to turn them into the representation the model expects.

That something is the transform pipeline.

The central idea of this chapter is:

The model does not train on your raw data. It trains on the output of your transforms.

That makes transforms part of the model’s effective input contract.

If the transform pipeline changes, the training problem may have changed even when the model code is identical.

That is why transforms deserve to be understood rather than copied.

1. The real input pipeline

A beginner often imagines training as:

image
  ↓
model
  ↓
loss

A more accurate picture is:

raw sample
    ↓
decode
    ↓
convert representation
    ↓
resize / crop / tokenize / scale
    ↓
normalize
    ↓
optional random augmentation
    ↓
model-ready tensor
    ↓
model
    ↓
loss

For images, that might be:

JPEG bytes
  ↓
RGB image
  ↓
[3,H,W] uint8 tensor
  ↓
resize / crop
  ↓
float32 in [0,1]
  ↓
normalize channels
  ↓
model

For text:

"the cat sat"
  ↓
tokenizer
  ↓
[17, 83, 41]
  ↓
pad / truncate
  ↓
input_ids + attention mask
  ↓
model

The details differ.

The principle does not.

A transform changes the representation presented to the model.

2. A transform is just a function

Forget TorchVision for a moment.

A transform can be as simple as:

import torch


def scale_to_unit_interval(x):
    return x.float() / 255.0


image = torch.randint(
    0,
    256,
    (3, 32, 32),
    dtype=torch.uint8,
)

transformed = scale_to_unit_interval(image)

print(image.dtype, image.min(), image.max())
print(transformed.dtype, transformed.min(), transformed.max())

Conceptually:

input
  ↓
transform
  ↓
output

Nothing more exotic is required.

A transform may:

change dtype
change shape
change value range
change spatial geometry
change tokenization
change labels
introduce randomness
remove information
add derived information

That list is why transform bugs can be so consequential.

3. Transform pipelines are composition

One transformation is rarely enough.

Suppose we want to:

convert image
resize it
cast to float
scale values
normalize channels

We can think of that as function composition:

raw
 ↓
T1
 ↓
T2
 ↓
T3
 ↓
T4
 ↓
model input

This connects directly to the recursive-composition idea from Step 04.

Complex behavior does not require one enormous transformation.

We compose simple transformations into a pipeline.

4. The current TorchVision transform API

For image work, TorchVision provides the torchvision.transforms.v2 API.

A basic classification pipeline can look like this:

import torch
from torchvision.transforms import v2


transform = v2.Compose([
    v2.Resize((224, 224)),
    v2.ToImage(),
    v2.ToDtype(torch.float32, scale=True),
    v2.Normalize(
        mean=[0.485, 0.456, 0.406],
        std=[0.229, 0.224, 0.225],
    ),
])

Then:

x = transform(raw_image)

The important object is not Compose itself.

It is the sequence of contracts:

raw image
  ↓
known spatial size
  ↓
image tensor
  ↓
float tensor with expected value scale
  ↓
normalized tensor

Every stage assumes something about what arrived from the stage before it.

5. ToImage() and ToDtype(..., scale=True) solve different problems

This distinction is worth understanding.

v2.ToImage()

means roughly:

represent this image as an image tensor.

Then:

v2.ToDtype(torch.float32, scale=True)

means:

convert the values to float32 and scale integer image intensities into the corresponding floating-point range.

These are separate operations because representation and numeric interpretation are separate concerns.

For a normal 8-bit image:

uint8 values:   0 ... 255

become approximately:

float values:   0.0 ... 1.0

That conversion looks trivial.

Getting it wrong can ruin training.

6. Dtype and value range are part of the data contract

Suppose the model expects:

float32 image in [0,1]

but receives:

float32 image in [0,255]

The dtype looks correct.

The shape looks correct.

The code runs.

The numerical meaning is wrong.

Inspect both:

def inspect_image(name, x):
    print(
        name,
        {
            "shape": tuple(x.shape),
            "dtype": str(x.dtype),
            "min": float(x.min()),
            "max": float(x.max()),
            "mean": float(x.float().mean()),
            "std": float(x.float().std()),
        },
    )

Use it before and after the transform:

inspect_image("raw", raw_tensor)
inspect_image("transformed", transformed_tensor)

Do not assume conversion did what you intended.

Observe it.

7. Order matters

Transforms are not usually commutative.

That means:

A then B

is not necessarily equivalent to:

B then A

For example, normalization expects floating-point values with a known interpretation.

A sensible image pipeline is:

convert representation
    ↓
convert dtype / scale values
    ↓
normalize

not:

normalize unknown uint8 representation
    ↓
hope later conversion fixes it

Similarly:

crop then resize

and:

resize then crop

can produce very different data distributions.

A transform pipeline is an ordered program.

Read it from top to bottom as carefully as model code.

8. What normalization actually does

A common transform is:

v2.Normalize(mean=mean, std=std)

For each channel, normalization applies the familiar operation:

x_normalized = (x - mean) / std

For an RGB image there are usually three channel statistics.

For example:

mean = [0.5, 0.5, 0.5]
std = [0.25, 0.25, 0.25]

Then each channel is transformed independently.

Normalization does not make an image “better”.

It changes the numeric coordinate system in which the model sees it.

That is why the chosen statistics are part of the model/data contract.

9. Never normalize blindly

Before normalization:

x = v2.ToDtype(torch.float32, scale=True)(image)

print(x.mean(dim=(1, 2)))
print(x.std(dim=(1, 2)))

After:

normalized = v2.Normalize(mean, std)(x)

print(normalized.mean(dim=(1, 2)))
print(normalized.std(dim=(1, 2)))

The exact values for one image will not become exactly zero mean and unit variance unless its channel statistics happen to match the reference statistics.

The transform is using dataset-level or pretrained-model reference statistics, not statistics calculated independently for every image.

That distinction matters.

10. Compute statistics from the training data, not from the future

Suppose you want dataset-specific normalization statistics.

Conceptually:

training split
    ↓
estimate mean/std
    ↓
freeze those numbers
    ↓
apply same transform to train and validation/test

Do not calculate preprocessing statistics using the validation or test set merely because those samples are available.

That leaks information across the evaluation boundary.

The same principle appears in tabular machine learning:

fit scaler on training data
apply fitted scaler to validation/test data

Transforms can leak information just as models can.

11. Training and validation transforms are often different

Training may deliberately introduce variation:

train_transform = v2.Compose([
    v2.RandomResizedCrop((224, 224), antialias=True),
    v2.RandomHorizontalFlip(p=0.5),
    v2.ToImage(),
    v2.ToDtype(torch.float32, scale=True),
    v2.Normalize(mean=mean, std=std),
])

Validation should usually be stable:

val_transform = v2.Compose([
    v2.Resize((256, 256), antialias=True),
    v2.CenterCrop((224, 224)),
    v2.ToImage(),
    v2.ToDtype(torch.float32, scale=True),
    v2.Normalize(mean=mean, std=std),
])

The difference is intentional:

TRAIN
random variation is part of learning

VALIDATION
stable measurement is the goal

If validation contains random augmentation, your evaluation metric can move because the benchmark itself changed from one pass to the next.

That connects directly to the regression/reproducibility chapter later in the book.

12. Augmentation changes the training distribution

Random augmentation is often described casually:

add some random crops and flips

But augmentation is a stronger claim than that.

You are saying:

these transformed examples should preserve the task-relevant meaning.

For an ordinary object photograph, a horizontal flip may preserve the label.

For text inside an image, road signs, handedness, medical scans or directional symbols, the same transformation may change meaning.

So augmentation should be chosen from the semantics of the problem.

Not from a list of popular transforms.

13. A transform can silently change the label meaning

Imagine classifying arrows:

← = class 0
→ = class 1

Now apply:

v2.RandomHorizontalFlip(p=1.0)

The image changes from left arrow to right arrow.

If the label stays unchanged, the training example is now contradictory.

The tensor is legal.

The shape is correct.

The transform executed successfully.

The dataset became wrong.

That is a much more dangerous failure than an exception.

14. Test stochastic transforms by applying them repeatedly

If a transform contains randomness, one output tells you very little.

Do this:

for i in range(5):
    out = train_transform(image)
    print(i, tuple(out.shape), out.mean().item(), out.std().item())

For visual data, plot several versions of the same source sample.

You want to ask:

Does the label still make sense?
Is important content routinely cropped away?
Is the augmentation too weak to matter?
Is it so strong that examples stop resembling the task?

Randomness should be inspected, not merely enabled.

15. transform and target_transform

Many TorchVision datasets accept separate callables:

dataset = SomeDataset(
    ...,
    transform=input_transform,
    target_transform=target_transform,
)

The distinction is straightforward:

transform
    changes the input / feature

target_transform
    changes the target / label

A target transform might map external string labels to integer IDs:

label_to_id = {
    "cat": 0,
    "dog": 1,
}


def target_transform(label):
    return label_to_id[label]

Do not transform targets simply because the option exists.

The target representation should follow the loss function’s contract.

16. Cross entropy usually wants class indices in our examples

Throughout this book we commonly use:

loss = torch.nn.functional.cross_entropy(logits, targets)

For ordinary single-label classification, our targets are class indices such as:

[2, 0, 1, 2, 1]

with integer dtype.

Do not automatically one-hot encode them into:

[[0,0,1],
 [1,0,0],
 ...]

because you saw a target_transform example somewhere.

First ask:

What representation does this loss expect for the problem I am solving?

Again, contracts before recipes.

17. Build a tiny dataset with an explicit transform boundary

A useful pattern is to keep raw sample acquisition separate from transformation.

from torch.utils.data import Dataset


class TinyImageDataset(Dataset):
    def __init__(self, images, labels, transform=None):
        self.images = images
        self.labels = labels
        self.transform = transform

    def __len__(self):
        return len(self.images)

    def __getitem__(self, index):
        image = self.images[index]
        label = self.labels[index]

        if self.transform is not None:
            image = self.transform(image)

        return image, label

Now the boundary is visible:

self.images[index]
        ↓
raw sample
        ↓
self.transform(image)
        ↓
model-ready sample

That boundary is one of the first places to inspect when training behaves strangely.

18. Make the contract executable

Suppose our CNN expects:

shape: [3,224,224]
dtype: float32
finite values

Write that down in code:

def require_model_image(x):
    if x.shape != (3, 224, 224):
        raise ValueError(
            f"expected [3,224,224], got {tuple(x.shape)}"
        )

    if x.dtype != torch.float32:
        raise TypeError(
            f"expected float32, got {x.dtype}"
        )

    if not torch.isfinite(x).all():
        raise ValueError("image contains NaN or Inf")

Then wrap the transform while debugging:

def checked_transform(raw):
    x = transform(raw)
    require_model_image(x)
    return x

A bad transform should fail near the transform boundary, not six layers into the network.

19. Shape alone is not enough

These two tensors can both have:

[3,224,224]

while representing very different inputs:

Tensor A: RGB, float32, approximately [0,1]
Tensor B: BGR, float32, approximately [0,255]

The model sees numbers.

It does not know which convention you intended.

That means a useful input contract contains more than shape:

axis meaning
channel order
dtype
value range
normalization
special values / padding semantics

This is the same lesson from Step 01, applied to real data.

20. Channel order mistakes survive many operations

Suppose an external image library returns:

[B,H,W,C]

but the model expects:

[B,C,H,W]

Or a library returns channels in BGR order while the model expects RGB.

The values are still valid numbers.

Some operations may even run.

Always attach semantic names:

3 = channels
224 = height
224 = width

and inspect known samples.

A transform pipeline is not correct merely because it produces the expected number of elements.

21. Visual inspection is a legitimate debugging tool

For image pipelines, show transformed examples.

If the tensor has been normalized, create a display copy rather than changing the training tensor.

For example:

def denormalize(x, mean, std):
    mean = torch.tensor(mean, device=x.device)[:, None, None]
    std = torch.tensor(std, device=x.device)[:, None, None]
    return x * std + mean

Then visualize:

preview = denormalize(x, mean, std).clamp(0, 1)

Do not train for six hours before discovering every image was accidentally almost black.

22. Paired data must transform together

Image classification is easy because the target is usually a class label.

Other tasks have structured targets.

For object detection:

image
+
bounding boxes
+
class labels

For segmentation:

image
+
pixel mask

If you crop or flip the image, the boxes or mask must undergo the corresponding geometric transformation.

Otherwise:

image says object is here
label says object is over there

TorchVision’s v2 transforms are designed to handle structured image targets such as images, bounding boxes and masks together.

The conceptual rule is:

Anything that shares geometry must share the random geometric decision.

23. The random crop must be the same random crop

Imagine doing this independently:

image = random_crop(image)
mask = random_crop(mask)

If each call chooses a different random crop, the pair is corrupted.

What we need is one sampled transformation applied consistently to related objects.

That principle matters beyond images.

For any paired or multimodal sample:

input A
input B
target

ask which transformations must preserve alignment.

24. Transform bugs can masquerade as model bugs

Suppose training loss refuses to fall.

You might suspect:

learning rate
optimizer
architecture
gradients
initialization

But the real bug may be:

labels no longer align with augmented inputs
normalization applied twice
wrong channel order
validation transform used for training
train transform used for validation
images still in 0..255
crop removes the object
padding token treated as normal input

This is why Step 08 begins its debugging ladder with the data.

The transform pipeline is part of “the data.”

25. A useful before/after sample inspector

Build a helper that exposes the boundary:

def inspect_transform(dataset, index, transform):
    raw = dataset[index]

    if isinstance(raw, tuple):
        raw_x, raw_y = raw
    else:
        raw_x, raw_y = raw, None

    transformed = transform(raw_x)

    print("RAW")
    print(" type:", type(raw_x).__name__)

    if torch.is_tensor(raw_x):
        print(" shape:", tuple(raw_x.shape))
        print(" dtype:", raw_x.dtype)

    print("TRANSFORMED")
    print(" type:", type(transformed).__name__)

    if torch.is_tensor(transformed):
        print(" shape:", tuple(transformed.shape))
        print(" dtype:", transformed.dtype)
        print(" min/max:", transformed.min().item(), transformed.max().item())

    print("TARGET:", raw_y)

The exact helper will vary by dataset.

The habit is what matters.

Inspect the representation before and after the transform.

26. Transform pipelines should have tests

Suppose your project depends on this invariant:

raw image
  ↓
[3,256,256] uint8
  ↓
transform
  ↓
[3,224,224] float32

Test it:

def test_train_transform_contract(sample_image):
    x = train_transform(sample_image)

    assert x.shape == (3, 224, 224)
    assert x.dtype == torch.float32
    assert torch.isfinite(x).all()

For a deterministic validation transform:

def test_val_transform_is_stable(sample_image):
    a = val_transform(sample_image)
    b = val_transform(sample_image)

    torch.testing.assert_close(a, b)

For a stochastic training transform, stability is not the expected property.

Test the invariants that should remain stable instead.

27. Test semantics, not only tensor legality

A transform can satisfy:

correct shape
correct dtype
finite values

and still be wrong for the task.

For important augmentation pipelines, keep a small gallery of known examples.

Ask a human question:

Does the transformed sample still have the target I say it has?

Some properties cannot be validated from tensor metadata alone.

28. Deterministic preprocessing and random augmentation are different things

It is useful to mentally split a transform pipeline into two parts.

Deterministic representation work

Examples:

decode
convert dtype
resize to required dimensions
tokenize with fixed tokenizer
normalize with fixed statistics
map categorical vocabulary

Stochastic augmentation

Examples:

random crop
random flip
color jitter
random masking
noise injection

That distinction helps with:

caching
reproducibility
validation
performance debugging

Deterministic work may be worth preprocessing or caching.

Random augmentation usually belongs where new variation can be sampled during training.

29. Where does the transform execute?

If your Dataset.__getitem__() applies the transform:

class DatasetWithTransform(Dataset):
    def __getitem__(self, index):
        x = self.load(index)
        x = self.transform(x)
        return x

then that transformation is part of sample production.

With DataLoader workers, much of that work can happen in worker processes.

That means a slow transform can become a data-loading bottleneck.

This is exactly why Step 05 told us to separate:

read time
transform time
collate time

Transforms are conceptually part of the data contract and operationally part of the input pipeline.

Both perspectives matter.

30. Profile the transform independently

Do not assume storage is the slow part.

Measure:

from time import perf_counter


def benchmark_transform(transform, samples):
    start = perf_counter()

    for sample in samples:
        transform(sample)

    elapsed = perf_counter() - start

    return {
        "samples": len(samples),
        "seconds": elapsed,
        "samples_per_second": len(samples) / elapsed,
    }

If transformation alone takes longer than the model step, more GPU optimization will not fix the pipeline.

31. Cache work that should not change

Suppose this is expensive:

read compressed source
  ↓
decode
  ↓
expensive deterministic feature extraction

and produces the same result every epoch.

Consider caching the deterministic representation.

Then keep random augmentation after the cache boundary:

raw
 ↓
expensive deterministic preprocessing
 ↓
CACHE
 ↓
random training augmentation
 ↓
model

This can dramatically change the performance characteristics of a training job.

But cache invalidation now becomes part of correctness.

If the transform definition changes, stale cached data may silently preserve the old behavior.

Version the cache or rebuild it deliberately.

32. Do not move every transform to the GPU blindly

Some tensor transforms can execute on accelerators.

That does not mean every transform should.

Moving preprocessing to the GPU changes:

CPU load
transfer volume
GPU utilization
synchronization
memory pressure
batching opportunities

The same rule from the performance chapter applies:

Measure the bottleneck before moving work.

A transform placement decision is a systems decision.

33. Reproducibility and random transforms

Random transforms mean two requests for the same dataset index may intentionally return different tensors.

That is useful during training.

It can also make debugging confusing.

When investigating one sample, temporarily control randomness or remove the random part of the pipeline.

For DataLoader workers, worker-specific random state also matters.

Step 05 already showed the pattern of seeding worker-local Python and NumPy RNGs from PyTorch’s worker seed.

The broader lesson is:

When randomness is intentional, record and control enough of it to reproduce failures when necessary.

34. A transform is not necessarily an image transform

The word “transform” often gets associated with image augmentation because TorchVision exposes a rich transform library.

But the concept is general.

For tabular data:

raw row
  ↓
parse numeric fields
  ↓
encode categories
  ↓
fill / represent missing values
  ↓
scale continuous features
  ↓
tensor

For text:

raw text
  ↓
normalize text if appropriate
  ↓
tokenize
  ↓
map tokens to IDs
  ↓
truncate / pad
  ↓
attention mask

For audio:

waveform
  ↓
resample
  ↓
window / feature representation
  ↓
normalize
  ↓
tensor

The model always receives the transformed representation.

That is the transferable concept.

35. A simple tabular transform from first principles

Suppose one sample contains:

sample = {
    "age": 41.0,
    "income": 72000.0,
}

We want standardized features using statistics learned from the training set.

class Standardize:
    def __init__(self, mean, std):
        self.mean = torch.tensor(mean, dtype=torch.float32)
        self.std = torch.tensor(std, dtype=torch.float32)

    def __call__(self, x):
        x = torch.tensor(x, dtype=torch.float32)
        return (x - self.mean) / self.std

Usage:

transform = Standardize(
    mean=[40.0, 60000.0],
    std=[12.0, 25000.0],
)

x = transform([sample["age"], sample["income"]])
print(x)

The machinery is different from TorchVision.

The principle is identical.

36. Fitted transforms have state

The previous transform contained:

mean
std

Those values came from somewhere.

That means the transform itself has state derived from training data.

Treat that state as part of the experiment.

Save it.

Version it.

Apply the same fitted transform at inference time.

A model trained with one preprocessing state and served with another is not the same system.

37. Training and inference must agree on deterministic preprocessing

Suppose training uses:

RGB
224×224
float32
scale to [0,1]
normalize with mean/std A

but inference uses:

BGR
256×256
float32
0..255
no normalization

The model weights can be loaded perfectly.

Inference will still be broken.

Deployment correctness includes reproducing the deterministic input contract.

Save preprocessing configuration alongside the model when necessary.

38. Pretrained models come with preprocessing assumptions

When using a pretrained model, the weights were trained under a particular input representation.

That may include assumptions about:

image size
resize strategy
channel order
value range
normalization statistics
interpolation

Do not copy only the network architecture and ignore its transform contract.

The weights learned from a particular representation.

Using a materially different one changes what the first layer receives.

39. A complete classification transform pair

Here is a clean pattern for ordinary image classification:

import torch
from torchvision.transforms import v2


MEAN = [0.485, 0.456, 0.406]
STD = [0.229, 0.224, 0.225]


train_transform = v2.Compose([
    v2.RandomResizedCrop(
        size=(224, 224),
        antialias=True,
    ),
    v2.RandomHorizontalFlip(p=0.5),
    v2.ToImage(),
    v2.ToDtype(torch.float32, scale=True),
    v2.Normalize(mean=MEAN, std=STD),
])


val_transform = v2.Compose([
    v2.Resize((256, 256), antialias=True),
    v2.CenterCrop((224, 224)),
    v2.ToImage(),
    v2.ToDtype(torch.float32, scale=True),
    v2.Normalize(mean=MEAN, std=STD),
])

Do not copy these exact augmentations and statistics into every project.

The pattern is what matters:

training = valid semantics + useful randomness + model contract
validation = deterministic representation + same model contract

40. Put transforms into datasets explicitly

For a TorchVision dataset:

from torchvision import datasets


train_dataset = datasets.CIFAR10(
    root="data",
    train=True,
    download=True,
    transform=train_transform,
)

val_dataset = datasets.CIFAR10(
    root="data",
    train=False,
    download=True,
    transform=val_transform,
)

Then DataLoader handles batching and scheduling:

from torch.utils.data import DataLoader


train_loader = DataLoader(
    train_dataset,
    batch_size=64,
    shuffle=True,
    num_workers=4,
)

val_loader = DataLoader(
    val_dataset,
    batch_size=64,
    shuffle=False,
    num_workers=4,
)

Now the responsibilities are clear:

Dataset
    knows how to produce one sample

Transform
    changes the sample representation

DataLoader
    schedules and batches samples

Model
    consumes the resulting tensors

That separation is worth preserving.

41. Inspect the first real batch

Before training:

images, labels = next(iter(train_loader))

print("images:", images.shape)
print("dtype:", images.dtype)
print("min/max:", images.min().item(), images.max().item())
print("mean:", images.mean().item())
print("std:", images.std().item())

print("labels:", labels.shape)
print("label dtype:", labels.dtype)
print("label range:", labels.min().item(), labels.max().item())

Then assert what the model expects:

assert images.ndim == 4
assert images.shape[1:] == (3, 224, 224)
assert images.dtype == torch.float32
assert labels.dtype == torch.int64
assert torch.isfinite(images).all()

This one checkpoint catches an enormous class of errors before the first optimizer step.

42. Compare train and validation contracts

The stochastic content may differ.

The final model-facing representation should remain compatible.

train_x, train_y = next(iter(train_loader))
val_x, val_y = next(iter(val_loader))

print(train_x.shape, train_x.dtype)
print(val_x.shape, val_x.dtype)

assert train_x.shape[1:] == val_x.shape[1:]
assert train_x.dtype == val_x.dtype

If the model receives different channel counts, dtypes or spatial contracts between training and validation, fix that before interpreting metrics.

43. Common failure: normalization applied twice

This can happen when one layer of the stack already scales/normalizes data and another transform does it again.

Symptoms may include:

unexpectedly tiny or huge values
training instability
poor transfer-learning results
activation statistics far from expectation

Inspect values at each boundary.

Do not infer from the transform names alone.

44. Common failure: validation uses random training augmentation

Bad:

train_dataset = Dataset(transform=train_transform)
val_dataset = Dataset(transform=train_transform)

when train_transform contains strong randomness.

Your validation metric is now partly measuring a fresh random view every time.

Use a stable validation transform unless stochastic evaluation is explicitly part of the experiment.

45. Common failure: train and inference transforms drift apart

Training script:

resize → scale → normalize A

Serving code six months later:

resize differently → normalize B

The checkpoint loads.

Predictions degrade.

Treat preprocessing as versioned code.

A useful experiment record includes transform configuration or a version identifier.

46. Common failure: augmentation destroys rare but important features

Strong crops can remove small objects.

Color transforms can erase subtle diagnostic cues.

Random masking can hide the only informative token span.

The average example may still look sensible.

Inspect edge cases deliberately.

The question is not:

Does augmentation make the dataset look more varied?

It is:

Does the augmentation preserve the information the target is supposed to describe?

47. Common failure: debugging transformed data through a DataLoader first

When something looks wrong, simplify.

Instead of starting with:

8 workers
prefetching
random transforms
batch collation
GPU transfer

start with:

raw = dataset_without_transform[index]
transformed = transform(raw)

One sample.

One process.

No batching.

Then add complexity back.

This is the same isolation strategy used throughout the book.

48. The transform debugging ladder

When a data pipeline produces suspicious training behavior, work in this order:

    flowchart TD
    A[Training input suspicious] --> B[Inspect one raw sample]
    B --> C[Apply transform directly]
    C --> D[Inspect shape / dtype / range / semantics]
    D --> E{Correct?}
    E -- No --> F[Find first transform that changes contract incorrectly]
    E -- Yes --> G[Repeat stochastic transform several times]
    G --> H[Verify target still matches]
    H --> I[Compare train vs validation pipeline]
    I --> J[Load one batch with num_workers=0]
    J --> K[Inspect collated batch]
    K --> L[Add workers / prefetching]
    L --> M[Profile transform cost]
    M --> N[Train]
  

In text:

1. inspect one raw sample
2. apply the transform directly
3. inspect shape, dtype, range and meaning
4. find the first bad transformation
5. repeat random transforms on the same sample
6. verify labels/targets still align
7. compare train and validation contracts
8. test one DataLoader batch with num_workers=0
9. only then add multiprocessing and performance tuning
10. profile transform cost separately

This is much faster than debugging a transform through a complete training run.

49. Build a transform report

A tiny helper can make transform experiments easier to compare:

from dataclasses import dataclass


@dataclass
class TensorContract:
    shape: tuple[int, ...]
    dtype: str
    finite: bool
    minimum: float
    maximum: float
    mean: float
    std: float


def tensor_contract(x):
    xf = x.float()

    return TensorContract(
        shape=tuple(x.shape),
        dtype=str(x.dtype),
        finite=bool(torch.isfinite(xf).all()),
        minimum=float(xf.min()),
        maximum=float(xf.max()),
        mean=float(xf.mean()),
        std=float(xf.std()),
    )

Then:

print(tensor_contract(train_transform(image)))
print(tensor_contract(val_transform(image)))

This does not replace looking at the sample.

It gives you evidence about the numerical representation.

50. Challenge: break the transform pipeline deliberately

Take a working image-classification pipeline and create these failures one at a time.

Break 1: remove scaling

Produce a float32 image whose values are still approximately 0..255.

Observe the resulting input statistics.

Break 2: normalize twice

Apply the same normalization a second time.

Inspect values before training.

Break 3: random validation

Use train_transform for validation.

Run validation repeatedly without changing the model.

How much does the metric move?

Break 4: destructive augmentation

Choose an augmentation that can change the label semantics.

Find concrete examples where the target becomes wrong.

Break 5: slow Python transform

Add an intentionally slow operation inside __getitem__().

Measure DataLoader throughput with:

num_workers = 0
1
2
4

Connect the result back to Step 05.

Break 6: train/inference drift

Train with one deterministic preprocessing pipeline and evaluate with a materially different one.

Observe how a perfectly valid checkpoint can become a broken system.

51. What you should now understand

A transform is not cosmetic preprocessing around the “real” model.

It is part of the program that defines the model’s input.

The important questions are:

What is the raw representation?
What does each transform change?
What order are transforms applied in?
What shape reaches the model?
What dtype reaches the model?
What numeric range reaches the model?
What semantics must remain invariant?
Which transforms are random?
Which transforms were fitted from training data?
Do training and validation share the same final contract?
Does inference reproduce deterministic preprocessing?
Where does the transform execute?
How expensive is it?

If you can answer those questions, data preprocessing stops being a bag of recipes.

It becomes something you can reason about.

Proficiency checkpoint

Before moving on, you should be able to:

[ ] write a callable transform
[ ] compose multiple transforms in a meaningful order
[ ] explain ToImage vs dtype/range conversion
[ ] inspect a transformed sample's shape, dtype and value range
[ ] build separate train and validation pipelines
[ ] explain why augmentation must preserve target semantics
[ ] keep structured targets aligned with geometric transforms
[ ] test the model-facing input contract
[ ] debug one sample before debugging a full DataLoader
[ ] identify deterministic vs stochastic preprocessing
[ ] avoid fitting preprocessing on validation/test data
[ ] reproduce deterministic preprocessing at inference time
[ ] profile transform cost as part of the data pipeline

That is the skill this chapter is trying to build.

Not memorizing a list of augmentation classes.

Where the book goes next

We now understand the entire path from stored sample to model input:

raw data
  ↓
Dataset
  ↓
Transforms
  ↓
DataLoader
  ↓
batch tensor
  ↓
model

The next chapter applies that contract to images in earnest.

We will start with tensors shaped like:

[B,C,H,W]

and follow what convolution actually does to:

channels
height
width

So the next step is:

CNN shape reasoning and debugging

Because once the transformed tensor enters the network, geometry becomes the model’s problem.