<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
	<channel>
		<title>Distillation From First Principles on Programmer.ie: Modern AI programming</title>
		<link>http://programmer.ie/books/distillation/</link>
		<description>Recent content in Distillation From First Principles on Programmer.ie: Modern AI programming</description>
		<generator>Hugo</generator>
		<language>en-US</language>
		
		
		
		
			<lastBuildDate>Mon, 16 Nov 2026 00:00:00 +0000</lastBuildDate>
		
			<atom:link href="http://programmer.ie/books/distillation/index.xml" rel="self" type="application/rss+xml" />
			<item>
				<title>What Is Knowledge Distillation?</title>
				<link>http://programmer.ie/books/distillation/01-chapter/</link>
				<pubDate>Thu, 12 Nov 2026 00:00:00 +0000</pubDate>
				<guid>http://programmer.ie/books/distillation/01-chapter/</guid>
				<description>&lt;p&gt;The first experiment in this book did not produce the easy result. A teacher&#xA;network classified a small synthetic problem almost perfectly: about 99.9% mean&#xA;held-out accuracy. A student with 228 parameters, against the teacher&amp;rsquo;s 17,412,&#xA;was trained in two ways on the same data: once on the task labels alone, and&#xA;once on the labels plus the teacher&amp;rsquo;s probabilities. Averaged over three seeds,&#xA;the student that learned from the teacher was the less accurate of the two. Two&#xA;of the three seeds favoured the label-only student; one favoured the student&#xA;that learned from the teacher.&lt;/p&gt;</description>
			</item>
			<item>
				<title>Distillation Versus Other AI Improvements</title>
				<link>http://programmer.ie/books/distillation/02-chapter/</link>
				<pubDate>Fri, 13 Nov 2026 00:00:00 +0000</pubDate>
				<guid>http://programmer.ie/books/distillation/02-chapter/</guid>
				<description>&lt;p&gt;A word that describes every improvement describes none. A model is called&#xA;distilled when it is smaller, trained on another model&amp;rsquo;s outputs or synthetic&#xA;data, quantised, merged with another checkpoint, or given a memory of tool calls.&#xA;These operations can all be useful. They are not the same operation; one name&#xA;hides the questions that determine whether they worked.&lt;/p&gt;&#xA;&lt;p&gt;The confusion is practical. A team that believes quantisation is distillation may&#xA;expect a smaller file to improve accuracy. A team that treats every synthetic&#xA;dataset as a distillation dataset may ignore the teacher&amp;rsquo;s errors or the terms&#xA;under which its outputs were produced. A team that treats teacher agreement as&#xA;proof of capability may deploy a student that has learned to imitate the&#xA;teacher&amp;rsquo;s mistakes.&lt;/p&gt;</description>
			</item>
			<item>
				<title>The Teacher and the Student</title>
				<link>http://programmer.ie/books/distillation/03-chapter/</link>
				<pubDate>Fri, 13 Nov 2026 00:00:00 +0000</pubDate>
				<guid>http://programmer.ie/books/distillation/03-chapter/</guid>
				<description>&lt;p&gt;Two models can each be excellent and still be useless as a pair. If token id 42&#xA;is &lt;code&gt;&amp;quot;cat&amp;quot;&lt;/code&gt; in the teacher&amp;rsquo;s vocabulary and &lt;code&gt;&amp;quot;the&amp;quot;&lt;/code&gt; in the student&amp;rsquo;s, a loss that&#xA;matches teacher logit 42 to student logit 42 compares different units, however&#xA;good either model is. If the teacher is reachable only through a service whose&#xA;terms restrict using its outputs to train a competing model, its accuracy does&#xA;not settle whether it can be used at all.&lt;/p&gt;</description>
			</item>
			<item>
				<title>Probabilities, Logits, and Soft Targets</title>
				<link>http://programmer.ie/books/distillation/04-chapter/</link>
				<pubDate>Fri, 13 Nov 2026 00:00:00 +0000</pubDate>
				<guid>http://programmer.ie/books/distillation/04-chapter/</guid>
				<description>&lt;p&gt;Give a softmax written directly from its textbook formula the three numbers&#xA;\((1000, 999, 998)\) and it returns not-a-number. Give it \((2, 1, 0)\) and it&#xA;returns \((0.6652, 0.2447, 0.0900)\). Yet the two inputs describe exactly the&#xA;same distribution, and a stable implementation returns those same three&#xA;probabilities for both. The numbers a classifier produces are not&#xA;probabilities; once they pass through a softmax, only their differences matter;&#xA;and how they are turned into a distribution is a design decision with numerical&#xA;consequences.&lt;/p&gt;</description>
			</item>
			<item>
				<title>Deriving the Distillation Objective</title>
				<link>http://programmer.ie/books/distillation/05-chapter/</link>
				<pubDate>Fri, 13 Nov 2026 00:00:00 +0000</pubDate>
				<guid>http://programmer.ie/books/distillation/05-chapter/</guid>
				<description>&lt;p&gt;Soften a teacher and a student far enough and the divergence between them&#xA;almost vanishes. In this chapter&amp;rsquo;s worked two-class example, the KL divergence&#xA;between the temperature-scaled teacher and student is \(0.2574\) at \(T=2\) and&#xA;\(0.0001125\) at \(T=100\). If that number were the training loss, raising the&#xA;temperature would shrink the teacher&amp;rsquo;s contribution towards zero. The classical&#xA;objective multiplies it by \(T^2\), which keeps it between \(1.0296\) and&#xA;\(1.1250\) at the temperatures the chapter checks. That factor, and the&#xA;direction in which the divergence is written, are not decoration. They are part&#xA;of what the student is trained to do.&lt;/p&gt;</description>
			</item>
			<item>
				<title>Response-Based and Logit Distillation</title>
				<link>http://programmer.ie/books/distillation/06-chapter/</link>
				<pubDate>Fri, 13 Nov 2026 00:00:00 +0000</pubDate>
				<guid>http://programmer.ie/books/distillation/06-chapter/</guid>
				<description>&lt;p&gt;Add 5 to every logit a student produces and nothing about its predictions&#xA;changes. In this chapter&amp;rsquo;s first worked example, the student&amp;rsquo;s probabilities&#xA;stay exactly where they were, and so does its scaled KL divergence from the&#xA;teacher, at 0.2606. Its raw-logit mean-squared error against the same teacher&#xA;rises from 0.75 to 24.0833. Both losses compare the same student with the same&#xA;teacher. They disagree because they are matching different representations of&#xA;the teacher&amp;rsquo;s output, and each representation keeps some information and&#xA;throws the rest away.&lt;/p&gt;</description>
			</item>
			<item>
				<title>Black-Box and API-Only Distillation</title>
				<link>http://programmer.ie/books/distillation/07-chapter/</link>
				<pubDate>Fri, 13 Nov 2026 00:00:00 +0000</pubDate>
				<guid>http://programmer.ie/books/distillation/07-chapter/</guid>
				<description>&lt;p&gt;In this chapter&amp;rsquo;s experiment, the training loop asked the teacher for 180,000&#xA;answers per arm and seed, and the teacher computed 1,200 of them. Both numbers&#xA;are correct. They count different things: logical requests and actual&#xA;evaluations after caching. A report that gave only one of them would misstate&#xA;the teacher&amp;rsquo;s workload by a factor of 150. Restricted access to a teacher is&#xA;like that throughout: what the student receives, what the teacher computes, and&#xA;what the provider permits are separate quantities, and each has to be recorded.&lt;/p&gt;</description>
			</item>
			<item>
				<title>Feature-Based Distillation</title>
				<link>http://programmer.ie/books/distillation/08-chapter/</link>
				<pubDate>Fri, 13 Nov 2026 00:00:00 +0000</pubDate>
				<guid>http://programmer.ie/books/distillation/08-chapter/</guid>
				<description>&lt;p&gt;Two students in this chapter&amp;rsquo;s experiment were trained to imitate the same&#xA;teacher layer in different ways. One matched the layer&amp;rsquo;s values; the other&#xA;matched only their directions. Their cosine distances from the teacher ended up&#xA;almost identical, 0.077 and 0.079, while their feature errors were 0.318 and&#xA;1.279. Both had slightly higher mean accuracy than a student&#xA;trained on labels alone, but by less than the 7–9 point spread between seeds.&#xA;Looking more like the teacher inside the network and doing better on the task&#xA;are separate outcomes, and this chapter measures them separately.&lt;/p&gt;</description>
			</item>
			<item>
				<title>Attention and Relational Distillation</title>
				<link>http://programmer.ie/books/distillation/09-chapter/</link>
				<pubDate>Sat, 14 Nov 2026 00:00:00 +0000</pubDate>
				<guid>http://programmer.ie/books/distillation/09-chapter/</guid>
				<description>&lt;p&gt;In this chapter&amp;rsquo;s experiment, a student trained to keep its teacher&amp;rsquo;s pairwise&#xA;distances finished with mean accuracy 5.63 percentage points above a student&#xA;trained on labels alone, the largest difference among the seven arms. Most of&#xA;that difference came from one of the three seeds. The distance error the student&#xA;was trained to reduce did fall, but that is what optimising it should do; it is&#xA;a consistency check, not evidence that the preserved structure caused the gain.&#xA;Relational supervision is promising here and fragile here, and this chapter&#xA;keeps both observations in view.&lt;/p&gt;</description>
			</item>
			<item>
				<title>Sequence-Level Distillation</title>
				<link>http://programmer.ie/books/distillation/10-chapter/</link>
				<pubDate>Sat, 14 Nov 2026 00:00:00 +0000</pubDate>
				<guid>http://programmer.ie/books/distillation/10-chapter/</guid>
				<description>&lt;p&gt;Students in this chapter&amp;rsquo;s experiment predicted the next symbol correctly&#xA;90.0–97.6% of the time when they were fed the correct prefix, yet produced the&#xA;whole correct sequence only 47–85% of the time, depending on seed, when left to&#xA;run on their own. Force a student&amp;rsquo;s first output symbol to be wrong and, in&#xA;every case checked, it never completed the sequence correctly. Sequence models&#xA;are trained under one condition and used under another. This chapter is about&#xA;what changes when both the supervision and the evaluation are whole sequences.&lt;/p&gt;</description>
			</item>
			<item>
				<title>Ensembles and Multiple Teachers</title>
				<link>http://programmer.ie/books/distillation/11-chapter/</link>
				<pubDate>Sat, 14 Nov 2026 00:00:00 +0000</pubDate>
				<guid>http://programmer.ie/books/distillation/11-chapter/</guid>
				<description>&lt;p&gt;Two teachers in this chapter&amp;rsquo;s experiment each classified more than 99% of the&#xA;test inputs correctly, and they disagreed with each other on only 0.6–1.1% of&#xA;them. Averaging them added almost nothing: on one seed the average was exactly&#xA;as accurate as the better teacher. Students trained on that average agreed with&#xA;it on about 88–89% of inputs, sat about 11 points below its accuracy, and could not&#xA;be told apart from a student trained on one fixed teacher. Two teachers are not&#xA;automatically more information than one, and this chapter is about measuring&#xA;when they are.&lt;/p&gt;</description>
			</item>
			<item>
				<title>The Data Behind the Knowledge</title>
				<link>http://programmer.ie/books/distillation/12-chapter/</link>
				<pubDate>Sun, 15 Nov 2026 00:00:00 +0000</pubDate>
				<guid>http://programmer.ie/books/distillation/12-chapter/</guid>
				<description>&lt;p&gt;One field in the E3 experiment&amp;rsquo;s manifest records 1,200 teacher-generated&#xA;targets. The experiment generated 3,600: 1,200 for each of three seeds, with the&#xA;field keeping only the count from the final seed&amp;rsquo;s loop. Nothing in a training&#xA;curve would reveal the difference. Only an audit of the records does, and an&#xA;audit can only find what the records kept. This chapter is about keeping the&#xA;records that make such an audit possible.&lt;/p&gt;</description>
			</item>
			<item>
				<title>Constructing Teaching Tasks</title>
				<link>http://programmer.ie/books/distillation/13-chapter/</link>
				<pubDate>Sun, 15 Nov 2026 00:00:00 +0000</pubDate>
				<guid>http://programmer.ie/books/distillation/13-chapter/</guid>
				<description>&lt;p&gt;Take a constructed prediction that gets seven of a sequence&amp;rsquo;s eight tokens&#xA;right. One metric scores it 0.875; another scores it 0.0: the single wrong&#xA;token sat at the&#xA;end-of-sequence position, so no valid output exists. Both numbers describe the&#xA;same prediction. Which of them counts is not a training decision. It is settled&#xA;earlier, when the teaching task is defined, and if it is settled wrongly no&#xA;amount of data will repair it.&lt;/p&gt;</description>
			</item>
			<item>
				<title>Synthetic Data Generation</title>
				<link>http://programmer.ie/books/distillation/14-chapter/</link>
				<pubDate>Sun, 15 Nov 2026 00:00:00 +0000</pubDate>
				<guid>http://programmer.ie/books/distillation/14-chapter/</guid>
				<description>&lt;p&gt;The generation study in this chapter produced 2,304 synthetic candidates in&#xA;under half a second, and about 95% of them passed verification. Neither number&#xA;says whether those candidates would teach anything. They were drawn from a space&#xA;of 87,296 possible inputs; the generator spread its draws evenly across&#xA;lengths, so the small short-length subspaces were sampled densely while the far&#xA;larger long-length spaces were barely touched; and no student was trained on&#xA;the result. Plenty is cheap; coverage,&#xA;correctness, and provenance are what have to be earned.&lt;/p&gt;</description>
			</item>
			<item>
				<title>Data Filtering and Curriculum Learning</title>
				<link>http://programmer.ie/books/distillation/15-chapter/</link>
				<pubDate>Sun, 15 Nov 2026 00:00:00 +0000</pubDate>
				<guid>http://programmer.ie/books/distillation/15-chapter/</guid>
				<description>&lt;p&gt;Chapter 14 generated 2,304 candidates and measured what generation&#xA;produces. This chapter asks what to do with them: which examples the&#xA;student should see, in what order, and what the selection rule silently&#xA;optimises. The central argument is:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;Filtering and curriculum learning are interventions on the teaching&#xA;distribution. They can remove noise, improve training efficiency, and&#xA;change learning dynamics — but they can also remove useful diversity,&#xA;overrepresent easy examples, and cause systematic failures on&#xA;underrepresented cases.&lt;/p&gt;</description>
			</item>
			<item>
				<title>Dataset Distillation</title>
				<link>http://programmer.ie/books/distillation/16-chapter/</link>
				<pubDate>Sun, 15 Nov 2026 00:00:00 +0000</pubDate>
				<guid>http://programmer.ie/books/distillation/16-chapter/</guid>
				<description>&lt;p&gt;Sixteen randomly chosen examples trained students to a mean held-out accuracy&#xA;of 0.756&#xA;in this chapter&amp;rsquo;s experiment, against 0.807 for the full 1,200.&#xA;Sixteen examples synthesised by dataset distillation reached 0.762, a&#xA;difference from the random sixteen that sits inside the seed noise; on two of&#xA;the three seeds the random subset did better. The method is real and the&#xA;mathematics is sound, but on this task it had almost nothing to improve on, and&#xA;the chapter reports that rather than a win.&lt;/p&gt;</description>
			</item>
			<item>
				<title>Distilling Transformers and Instruction Models</title>
				<link>http://programmer.ie/books/distillation/17-chapter/</link>
				<pubDate>Sun, 15 Nov 2026 00:00:00 +0000</pubDate>
				<guid>http://programmer.ie/books/distillation/17-chapter/</guid>
				<description>&lt;p&gt;Three groups of transformer students in this chapter&amp;rsquo;s experiment, three seeds&#xA;each, reached 0.525, 0.528, and 0.525 mean exact match. The first learned from ground-truth labels, the&#xA;second from a teacher&amp;rsquo;s soft token distributions, and the third from the&#xA;teacher&amp;rsquo;s own generated outputs. The third matched the first to the third&#xA;decimal on every seed, and not by coincidence: the teacher was perfect, so its&#xA;outputs were the labels. A distillation result means little until it is set&#xA;beside the plain supervised baseline and the reason for any tie is understood.&lt;/p&gt;</description>
			</item>
			<item>
				<title>Distilling Reasoning</title>
				<link>http://programmer.ie/books/distillation/18-chapter/</link>
				<pubDate>Sun, 15 Nov 2026 00:00:00 +0000</pubDate>
				<guid>http://programmer.ie/books/distillation/18-chapter/</guid>
				<description>&lt;p&gt;Trace-trained students in this chapter&amp;rsquo;s experiment predicted the next token of a worked&#xA;arithmetic trace correctly about 90% of the time when they were fed the correct&#xA;trace so far. Left to write the trace themselves, they produced a fully valid&#xA;one only 2–6% of the time. Knowing each step when shown the way there is not&#xA;the same as being able to walk it, and a reasoning claim has to rest on the&#xA;second.&lt;/p&gt;</description>
			</item>
			<item>
				<title>Verifier-Guided Distillation</title>
				<link>http://programmer.ie/books/distillation/19-chapter/</link>
				<pubDate>Sun, 15 Nov 2026 00:00:00 +0000</pubDate>
				<guid>http://programmer.ie/books/distillation/19-chapter/</guid>
				<description>&lt;p&gt;Two students in this chapter&amp;rsquo;s experiment were trained on traces selected by&#xA;two different verifiers: one that checked only the final answer, and one that&#xA;re-executed every step. Their results matched to three decimals on every seed.&#xA;The verifiers had agreed on all 2,400 candidates, because every teacher trace&#xA;with a correct answer here also had valid steps. A process verifier can only&#xA;show its value where answers and reasoning come apart, and this chapter&#xA;measures whether they did.&lt;/p&gt;</description>
			</item>
			<item>
				<title>On-Policy Distillation</title>
				<link>http://programmer.ie/books/distillation/20-chapter/</link>
				<pubDate>Mon, 16 Nov 2026 00:00:00 +0000</pubDate>
				<guid>http://programmer.ie/books/distillation/20-chapter/</guid>
				<description>&lt;p&gt;Chapters 17 through 19 trained the student on fixed collections: oracle&#xA;answers, oracle traces, teacher-generated candidates, verifier-selected&#xA;subsets. The training states never changed, no matter how the student&#xA;changed. This chapter asks what happens when they do:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;Does supervising the student on its own generated outputs improve the&#xA;match between training and inference distributions, and at what cost?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;The honest answer, on our task, is two-sided. Matching the prefix&#xA;distribution does nothing measurable for correctness — the on-policy&#xA;student ties the offline student to three decimals (0.452 versus&#xA;0.453). But the choice is not irrelevant: training on samples drawn once&#xA;from the untrained initial student and replayed throughout collapses&#xA;correctness from 0.45 to 0.08. Those samples were both stale and&#xA;near-garbage, so the experiment cannot say which of the two did the&#xA;damage. Where the states come from matters enormously when they are&#xA;wrong and stale, and not measurably, here, when they are fresh. That asymmetry is the chapter&amp;rsquo;s&#xA;finding, and the machinery below exists to make it checkable rather&#xA;than merely plausible.&lt;/p&gt;</description>
			</item>
			<item>
				<title>Self-Distillation</title>
				<link>http://programmer.ie/books/distillation/21-chapter/</link>
				<pubDate>Mon, 16 Nov 2026 00:00:00 +0000</pubDate>
				<guid>http://programmer.ie/books/distillation/21-chapter/</guid>
				<description>&lt;p&gt;Chapter 20 changed &lt;em&gt;where&lt;/em&gt; training states come from. This chapter&#xA;changes &lt;em&gt;who&lt;/em&gt; the teacher is: an earlier generation of the student&#xA;itself. The question is sharper than it looks:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;Can a model improve by learning from a version of itself, and when&#xA;is that just self-training or confirmation bias?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;The chapter&amp;rsquo;s answer, measured on a four-class spiral with tiny&#xA;MLPs, is that the words matter more than the weights. Soft&#xA;supervision from a flawed same-arch teacher is associated with a small&#xA;mean gain (+0.032 held-out accuracy, below the baseline on one of three&#xA;seeds). Hard pseudo-labels from the same teacher, mixed with ground&#xA;truth, give a similar mean gain (+0.022, also below the baseline on one&#xA;seed) while visibly importing the teacher&amp;rsquo;s confident errors.&lt;/p&gt;</description>
			</item>
			<item>
				<title>Preference and Alignment Distillation</title>
				<link>http://programmer.ie/books/distillation/22-chapter/</link>
				<pubDate>Mon, 16 Nov 2026 00:00:00 +0000</pubDate>
				<guid>http://programmer.ie/books/distillation/22-chapter/</guid>
				<description>&lt;p&gt;Chapters 17 through 21 supervised the student with right answers:&#xA;oracle completions, oracle traces, verified traces, the student&amp;rsquo;s own&#xA;trajectories scored by a teacher, an earlier generation&amp;rsquo;s soft&#xA;targets. This chapter changes the supervision signal itself. No one&#xA;tells the student the right answer any more; someone only tells it&#xA;which of two answers was preferred:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;Can a student learn preferred behaviour without confusing teacher&#xA;preferences with independently correct answers?&lt;/p&gt;</description>
			</item>
			<item>
				<title>Continual and Adaptive Distillation</title>
				<link>http://programmer.ie/books/distillation/23-chapter/</link>
				<pubDate>Mon, 16 Nov 2026 00:00:00 +0000</pubDate>
				<guid>http://programmer.ie/books/distillation/23-chapter/</guid>
				<description>&lt;p&gt;Chapters 17 through 22 trained a student once, on one distribution,&#xA;and evaluated it once. Real deployments do not work that way:&#xA;teachers are refreshed, data arrives in batches, tasks change, and&#xA;someone eventually asks for an example to be deleted. This chapter&#xA;asks the operational question:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;How can a student keep learning from new teachers or data without&#xA;forgetting what it already knew — and what must be retained or&#xA;deleted?&lt;/p&gt;</description>
			</item>
			<item>
				<title>Classification, Embeddings, and Retrieval</title>
				<link>http://programmer.ie/books/distillation/24-chapter/</link>
				<pubDate>Mon, 16 Nov 2026 00:00:00 +0000</pubDate>
				<guid>http://programmer.ie/books/distillation/24-chapter/</guid>
				<description>&lt;p&gt;A classifier chooses one label from a fixed set. A retriever ranks a&#xA;corpus whose size is not fixed, for a query it has never seen. The&#xA;difference sounds organisational until the teacher arrives: a&#xA;cross-encoder can read a query and a document together and score the&#xA;pair, but it must do that for every document at query time. A&#xA;bi-encoder must turn each side into a vector once, then compare&#xA;vectors. The compact student is therefore not merely smaller; it is a&#xA;different kind of function:&lt;/p&gt;</description>
			</item>
			<item>
				<title>Coding and Structured Generation</title>
				<link>http://programmer.ie/books/distillation/25-chapter/</link>
				<pubDate>Mon, 16 Nov 2026 00:00:00 +0000</pubDate>
				<guid>http://programmer.ie/books/distillation/25-chapter/</guid>
				<description>&lt;p&gt;Chapter 24 ranked documents. The output was an ordering, and the&#xA;evaluation asked where the relevant document landed. This chapter&#xA;changes the output object again. A program, a SQL query, or a JSON&#xA;document is not liked or ranked; it runs, parses, or fails. Its&#xA;correctness is a property of the artefact, checkable without asking&#xA;any model for its opinion:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;Can a student learn to generate functionally correct structured&#xA;outputs from teacher demonstrations rather than merely copying the&#xA;teacher&amp;rsquo;s syntax?&lt;/p&gt;</description>
			</item>
			<item>
				<title>Tool Use and Agent Trajectories</title>
				<link>http://programmer.ie/books/distillation/26-chapter/</link>
				<pubDate>Mon, 16 Nov 2026 00:00:00 +0000</pubDate>
				<guid>http://programmer.ie/books/distillation/26-chapter/</guid>
				<description>&lt;p&gt;Chapter 25 generated an output and checked it. The whole task was one&#xA;decision. An agent faces a different structure: it chooses a tool,&#xA;reads the observation, chooses again, and only after several decisions&#xA;produces anything checkable. Each decision changes what the next&#xA;decision sees:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;Can a student learn to complete multi-step tasks by distilling a&#xA;teacher&amp;rsquo;s actions and tool-use trajectories?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;On our bounded study, the measured answer is no — and the way it&#xA;fails is the chapter&amp;rsquo;s real subject. Three policies, three seeds, 108&#xA;held-out episodes: zero successes. Yet the same policies agree with&#xA;the teacher on roughly 30% of held-out teacher states, and the&#xA;recovery-augmented student answers a third of forced error states&#xA;correctly while the plain cloning student answers none. Imitation&#xA;without generalisation, local robustness without task success,&#xA;agreement without completion. The trajectory experiment exists to&#xA;keep those apart.&lt;/p&gt;</description>
			</item>
			<item>
				<title>Distilling Memory and Procedural Skills</title>
				<link>http://programmer.ie/books/distillation/27-chapter/</link>
				<pubDate>Mon, 16 Nov 2026 00:00:00 +0000</pubDate>
				<guid>http://programmer.ie/books/distillation/27-chapter/</guid>
				<description>&lt;p&gt;Chapter 26 ended with a student that reproduces training rollouts and&#xA;fails new situations. The diagnosis offered, as a hypothesis, was that&#xA;the policy memorised training states without extracting the rule that&#xA;would generalise; the floor result was consistent with that reading&#xA;without isolating it. This chapter asks a related question: whether&#xA;the facts an agent answers from have to live in its weights at all,&#xA;and tests whether&#xA;that knowledge has to live in the weights at all:&lt;/p&gt;</description>
			</item>
			<item>
				<title>Multi-Agent and Expert Distillation</title>
				<link>http://programmer.ie/books/distillation/28-chapter/</link>
				<pubDate>Mon, 16 Nov 2026 00:00:00 +0000</pubDate>
				<guid>http://programmer.ie/books/distillation/28-chapter/</guid>
				<description>&lt;p&gt;Chapters 26 and 27 distilled one teacher — scripted, optimal, alone.&#xA;Real deployments rarely look like that. Knowledge lives in several&#xA;places: one model that handles one region well, another that handles&#xA;the rest, a router that decides who answers. The question is what&#xA;&amp;ldquo;distilling&amp;rdquo; those teachers means, and what gets lost when several&#xA;behaviours are aggregated into one student:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;Can two specialised teachers transfer their complementary&#xA;capabilities to one student, and what is lost when their behaviour&#xA;is aggregated?&lt;/p&gt;</description>
			</item>
			<item>
				<title>Vision, Speech, and Multimodal Distillation</title>
				<link>http://programmer.ie/books/distillation/29-chapter/</link>
				<pubDate>Mon, 16 Nov 2026 00:00:00 +0000</pubDate>
				<guid>http://programmer.ie/books/distillation/29-chapter/</guid>
				<description>&lt;p&gt;Chapter 28 routed between experts that shared an observation space.&#xA;This chapter removes that comfort: the teacher sees images, the&#xA;student hears only signals — sixteen numbers that encode the same&#xA;four classes through a different sense. Nothing lines up by&#xA;construction. The dimensions differ, the invariances differ, and no&#xA;coordinate in one space means anything in the other:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;What information can be transferred between modalities, and when&#xA;does preserving an embedding relationship fail to preserve the&#xA;underlying capability?&lt;/p&gt;</description>
			</item>
			<item>
				<title>Distillation for Generative Media</title>
				<link>http://programmer.ie/books/distillation/30-chapter/</link>
				<pubDate>Mon, 16 Nov 2026 00:00:00 +0000</pubDate>
				<guid>http://programmer.ie/books/distillation/30-chapter/</guid>
				<description>&lt;p&gt;Every chapter so far distilled a decision: a label, a ranking, a&#xA;JSON document, an action, a fact from a store. This chapter distils&#xA;a distribution. A diffusion teacher does not know the answer to any&#xA;query; it knows how to turn noise into samples, one small denoising&#xA;step at a time — dozens or hundreds of them. The student question is&#xA;different from everything before it: not &amp;ldquo;can you be right more&#xA;cheaply&amp;rdquo; but &amp;ldquo;can you draw from the same distribution in fewer&#xA;steps, and what is lost when you do?&amp;rdquo;&lt;/p&gt;</description>
			</item>
			<item>
				<title>Choosing the Right Student</title>
				<link>http://programmer.ie/books/distillation/31-chapter/</link>
				<pubDate>Mon, 16 Nov 2026 00:00:00 +0000</pubDate>
				<guid>http://programmer.ie/books/distillation/31-chapter/</guid>
				<description>&lt;p&gt;Every experiment so far asked whether the student learned. This&#xA;chapter asks whether the student is worth deploying — a different&#xA;question with a different kind of answer. Learning is measured&#xA;against ground truth; deployment is measured against constraints. A&#xA;model that scores highest and misses its latency budget by a factor&#xA;of ten is not the best student. It is a failed deployment:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;Which student is the right engineering choice for a stated task,&#xA;budget, and deployment environment?&lt;/p&gt;</description>
			</item>
			<item>
				<title>Parameter-Efficient Distillation</title>
				<link>http://programmer.ie/books/distillation/32-chapter/</link>
				<pubDate>Mon, 16 Nov 2026 00:00:00 +0000</pubDate>
				<guid>http://programmer.ie/books/distillation/32-chapter/</guid>
				<description>&lt;p&gt;Chapter 31 chose among students by measured deployment cost and&#xA;found that distillation never changes the price tag: a student&amp;rsquo;s&#xA;architecture fixes its cost, and that choice is made before training&#xA;starts. This chapter asks the complementary question. Suppose&#xA;training itself is the scarce resource — optimiser memory, update&#xA;budget, the number of weights you may touch — while deployment cost&#xA;is fixed. Can the student acquire the teacher&amp;rsquo;s behaviour while&#xA;updating far fewer parameters?&lt;/p&gt;</description>
			</item>
			<item>
				<title>Distillation, Quantisation, and Compression</title>
				<link>http://programmer.ie/books/distillation/33-chapter/</link>
				<pubDate>Mon, 16 Nov 2026 00:00:00 +0000</pubDate>
				<guid>http://programmer.ie/books/distillation/33-chapter/</guid>
				<description>&lt;p&gt;Chapter 31 chose the student. Chapter 32 shrank the update. This&#xA;chapter shrinks what is stored and executed — or rather, it asks&#xA;which shrinking does what, because &amp;ldquo;compression&amp;rdquo; is four different&#xA;interventions wearing one name, and only some of them change the&#xA;quantities a deployment pays for:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;When distillation is combined with quantisation or pruning, how&#xA;much storage, memory, and runtime cost is actually removed, and&#xA;what capability is lost?&lt;/p&gt;</description>
			</item>
			<item>
				<title>Evaluating Knowledge Transfer</title>
				<link>http://programmer.ie/books/distillation/34-chapter/</link>
				<pubDate>Mon, 16 Nov 2026 00:00:00 +0000</pubDate>
				<guid>http://programmer.ie/books/distillation/34-chapter/</guid>
				<description>&lt;p&gt;Every experiment in this book reports two numbers that look alike and&#xA;mean different things: how often the student is right, and how often&#xA;the student agrees with the teacher. Thirty-three chapters have kept&#xA;them in separate columns. This chapter explains why that separation&#xA;is the whole of evaluation — and what else must accompany it before&#xA;anyone may claim that a capability transferred:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;What evidence justifies the claim that a student acquired a&#xA;capability from a teacher, rather than merely fitting a dataset or&#xA;copying a teacher?&lt;/p&gt;</description>
			</item>
			<item>
				<title>Failure Modes and Safety Regressions</title>
				<link>http://programmer.ie/books/distillation/35-chapter/</link>
				<pubDate>Mon, 16 Nov 2026 00:00:00 +0000</pubDate>
				<guid>http://programmer.ie/books/distillation/35-chapter/</guid>
				<description>&lt;p&gt;Chapter 34 taught the reader to measure what transferred. This&#xA;chapter teaches the reader to catch what broke — because the&#xA;average that proves a transfer can hide the slice that forbids a&#xA;deployment:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;How can a distilled student look better on average while being&#xA;worse, less safe, or less lawful in the cases that matter?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;The answer is a regression suite with teeth: slice-level&#xA;acceptance criteria, invalid-action detection, calibration checks,&#xA;deletion verification, teacher-error probes, and a rights gate —&#xA;combined by a promotion rule under which a single blocking finding&#xA;rejects the release no matter what the average says. Six&#xA;deterministic fixtures demonstrate the machinery; the book&amp;rsquo;s own&#xA;failed students show what the gate would have done to them.&lt;/p&gt;</description>
			</item>
			<item>
				<title>Model Licences, Contracts, and Permitted Use</title>
				<link>http://programmer.ie/books/distillation/36-chapter/</link>
				<pubDate>Mon, 16 Nov 2026 00:00:00 +0000</pubDate>
				<guid>http://programmer.ie/books/distillation/36-chapter/</guid>
				<description>&lt;p&gt;Thirty-five chapters treated permission as a gate: a record, a&#xA;status, a reviewer, a date. This chapter opens the gate and reads&#xA;what is inside — because &amp;ldquo;the teacher is available&amp;rdquo; never answered&#xA;the question the engineer must ask:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;What exactly does the recorded agreement permit, and what must be&#xA;escalated before a distillation run begins?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;The answer is always action-specific. A source may permit&#xA;evaluation but forbid redistribution, allow fine-tuning but forbid&#xA;competing-model development, or grant everything except commercial&#xA;use. The chapter builds that discipline into a licence-comparison&#xA;matrix, applies it to every E1–E4 study across twelve actions, and&#xA;works five hypothetical cases that are labelled as hypothetical —&#xA;because the most dangerous sentence in this territory is &amp;ldquo;it was&#xA;publicly downloadable, so we assumed&amp;rdquo;.&lt;/p&gt;</description>
			</item>
			<item>
				<title>Copyright and Training-Data Rights</title>
				<link>http://programmer.ie/books/distillation/37-chapter/</link>
				<pubDate>Mon, 16 Nov 2026 00:00:00 +0000</pubDate>
				<guid>http://programmer.ie/books/distillation/37-chapter/</guid>
				<description>&lt;p&gt;Chapter 36 asked what agreements permit. This chapter asks what the&#xA;law permits when no agreement exists — because teaching data is&#xA;where copyright enters the distillation pipeline, and the most&#xA;expensive mistake in this territory is assuming that material which&#xA;was lawful to read was therefore lawful to train on:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;Which rights attach to the data and outputs used in a&#xA;distillation pipeline, and which exceptions or permissions apply&#xA;in the relevant jurisdiction?&lt;/p&gt;</description>
			</item>
			<item>
				<title>Privacy, GDPR, and AI Regulation</title>
				<link>http://programmer.ie/books/distillation/38-chapter/</link>
				<pubDate>Mon, 16 Nov 2026 00:00:00 +0000</pubDate>
				<guid>http://programmer.ie/books/distillation/38-chapter/</guid>
				<description>&lt;p&gt;Chapter 37 asked who may copy. This chapter asks who may be&#xA;identified — because personal data, once inside a pipeline, follows&#xA;every artifact downstream, and no copyright permission establishes&#xA;a data-protection basis:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;How should personal-data rights, privacy obligations and AI&#xA;regulatory requirements influence a distillation system&amp;rsquo;s design,&#xA;especially when demonstrations, memory and model weights may&#xA;retain information?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;The answer separates three duties that share a pipeline but not a&#xA;logic: data protection for people, documentation for systems, and&#xA;deletion that must reach every copy including the copies that&#xA;cannot be deleted. This chapter is educational, not legal advice.&#xA;It states texts with versions and dates, marks interpretation, and&#xA;escalates everything structural. Nothing here certifies compliance.&lt;/p&gt;</description>
			</item>
			<item>
				<title>Building a Legally Auditable Distillation Pipeline</title>
				<link>http://programmer.ie/books/distillation/39-chapter/</link>
				<pubDate>Mon, 16 Nov 2026 00:00:00 +0000</pubDate>
				<guid>http://programmer.ie/books/distillation/39-chapter/</guid>
				<description>&lt;p&gt;Chapters 36–38 decided what is permitted, what is personal, and&#xA;what is required. This chapter builds the machine that traces any&#xA;of it — because a permission nobody can trace is a rumour, and a&#xA;measurement nobody can reproduce is a story:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;Can an independent reviewer trace a student from its sources and&#xA;permissions to its evaluation and promotion decision?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;The answer is a lineage graph with eight relations, a completeness&#xA;score that fails loudly, an integration audit over all twenty-five&#xA;book experiments, and a promotion summary that Chapters 40 and 42&#xA;consume directly. The audit&amp;rsquo;s verdict on this programme is&#xA;BLOCKED — and the chapter explains why a blocked audit is the&#xA;system working, not the system failing.&lt;/p&gt;</description>
			</item>
			<item>
				<title>Architecting a Distillation Engine</title>
				<link>http://programmer.ie/books/distillation/40-chapter/</link>
				<pubDate>Mon, 16 Nov 2026 00:00:00 +0000</pubDate>
				<guid>http://programmer.ie/books/distillation/40-chapter/</guid>
				<description>&lt;p&gt;Thirty-nine chapters built the parts: specifications, datasets,&#xA;losses, metrics, teachers, students, evaluation, rights gates,&#xA;safety checks, lineage, and audits. This chapter assembles them&#xA;into the smallest system that can run a distillation workflow&#xA;without lying about what it did:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;What is the smallest architecture that can run a lawful,&#xA;reproducible distillation experiment end to end?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;The answer is a state machine with eleven transitions and no&#xA;platform: a workbench that wires existing modules, refuses to&#xA;train without permission, replays frozen evidence without&#xA;relabelling it, and holds — rather than promotes — when a human&#xA;has not authorised the outcome. The workbench is a reference&#xA;learning system, not a product platform, and the chapter defends&#xA;that boundary explicitly.&lt;/p&gt;</description>
			</item>
			<item>
				<title>Four Controlled Distillation Experiments</title>
				<link>http://programmer.ie/books/distillation/41-chapter/</link>
				<pubDate>Mon, 16 Nov 2026 00:00:00 +0000</pubDate>
				<guid>http://programmer.ie/books/distillation/41-chapter/</guid>
				<description>&lt;p&gt;Forty chapters built the instruments. This one reads the&#xA;measurements — all of them, in one place, with the failures kept&#xA;in and the denominators attached:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;What do the four experiments jointly establish, and what remains&#xA;unmeasured?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;The short answer first, because the chapter earns it row by row:&#xA;distillation reliably transfers &lt;em&gt;fidelity&lt;/em&gt; — students learn to&#xA;resemble their teachers on the agreement measures in most families&#xA;tested — while&#xA;&lt;em&gt;capability&lt;/em&gt; transfers only under narrow, named conditions, and&#xA;fails outright in most agentic settings tried here. That sentence&#xA;is the book&amp;rsquo;s scientific conclusion in miniature. Everything&#xA;below is the evidence that constrains it, the uncertainty that&#xA;bounds it, and the list of what would be needed to say more.&lt;/p&gt;</description>
			</item>
			<item>
				<title>Deploying a Continuously Improving AI</title>
				<link>http://programmer.ie/books/distillation/42-chapter/</link>
				<pubDate>Mon, 16 Nov 2026 00:00:00 +0000</pubDate>
				<guid>http://programmer.ie/books/distillation/42-chapter/</guid>
				<description>&lt;p&gt;Forty-one chapters established what the students learned, what&#xA;they cost, what risks they carry, and whether their use is&#xA;authorised. One question remains, and it is the only one that&#xA;matters outside the laboratory:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;How could a distilled model be deployed and improved&#xA;continuously without silently degrading correctness, safety,&#xA;provenance, rights, or human control?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;The answer is a gate that can say no, monitoring that can change&#xA;its mind, a rollback that restores the past, and humans who own&#xA;every step. This chapter introduces &lt;strong&gt;KD-036 Promotion gate&lt;/strong&gt;&#xA;(registry: pre-defined criteria a student must meet before&#xA;deployment, covering correctness, safety, rights, and&#xA;reproducibility) — and demonstrates it entirely by simulation.&#xA;No live deployment is authorised, none is performed, and nothing&#xA;here promotes the book&amp;rsquo;s own students.&lt;/p&gt;</description>
			</item>
	</channel>
</rss>
