pytorch-from-first-principles examples on Programmer.ie: Modern AI programming

The loss fell to 11.25 and the model learned nothing

Mon, 01 Jan 0001 00:00:00 +0000

Broadcasting is a feature that is willing to make sense of shape combinations you did not intend to write. In the chapter’s opening program the targets arrive as a column, the predictions as a row, and the subtraction silently produces a four by four grid of errors: every prediction compared against every target. Nothing raises, the loss falls by two thirds, and the model ends up answering y = 7.5 whatever the input.

The loss went to zero and half the model never moved

Mon, 01 Jan 0001 00:00:00 +0000

While the tensors of a forward pass flow, PyTorch writes down a record of which operation produced which result, with the references needed to differentiate the computation. A missing gradient is a fact about that record.

The chapter’s opening loop detaches a feature before returning it, so the log can hold values without holding the graph. The loss still falls to zero, because the second parameter absorbs the whole job. This program composes the chapter’s pieces into one run: the broken loop, the record, a reachability check and the repair.

The error names the classifier; the trace names the convolution

Mon, 01 Jan 0001 00:00:00 +0000

A convolution changes two independent things: the channel count, which you declare, and the spatial extent, which you derive from the kernel, stride, padding and dilation. Get either wrong and the error arrives downstream, naming a layer that did nothing wrong.

This program composes the chapter’s pieces into one run: the derived rule checked against PyTorch over thousands of combinations, then a small network traced against a hand-written shape list, once as written and once with one padding removed.

Two tensors, the same shape, and not the same tensor

Mon, 01 Jan 0001 00:00:00 +0000

Attention is a sequence of explicit relationships between represented vectors, and almost every stage can be wrong while producing exactly the shape you predicted. Splitting the embedding across heads is the smallest example: a reshape gives the shape you wanted and not the arrangement of values you meant.

This program composes the chapter’s pieces into one run: the good and bad head split, a round trip that catches the bad one, a hand-written attention compared against PyTorch’s, and a future-token intervention.