<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
	<channel>
		<title>Inference From First Principles on Programmer.ie: Modern AI programming</title>
		<link>http://programmer.ie/examples/inference/</link>
		<description>Recent content in Inference From First Principles on Programmer.ie: Modern AI programming</description>
		<generator>Hugo</generator>
		<language>en-US</language>
		
		
		
		
			<atom:link href="http://programmer.ie/examples/inference/index.xml" rel="self" type="application/rss+xml" />
			<item>
				<title>Follow one request through every inference stage</title>
				<link>http://programmer.ie/examples/inference/01-chapter/</link>
				<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
				<guid>http://programmer.ie/examples/inference/01-chapter/</guid>
				<description>&lt;p&gt;A request is not one operation. It is parsed, tokenised, encoded, run through a&#xA;model many times, decoded and formatted, and each stage has a different owner&#xA;and a different failure mode. This example makes those boundaries visible with a&#xA;generator small enough to read in one sitting. Its &amp;ldquo;forward pass&amp;rdquo; is a seeded&#xA;table lookup, not a neural network, so the timings measure the surrounding&#xA;pipeline rather than a model.&lt;/p&gt;</description>
			</item>
			<item>
				<title>Find what changed in a model checkpoint</title>
				<link>http://programmer.ie/examples/inference/02-chapter/</link>
				<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
				<guid>http://programmer.ie/examples/inference/02-chapter/</guid>
				<description>&lt;p&gt;A model file is an artefact; an executing model is a process. Between them sit&#xA;three switches that are easy to conflate: the module&amp;rsquo;s training/eval mode, the&#xA;&lt;code&gt;requires_grad&lt;/code&gt; flag on its parameters, and PyTorch&amp;rsquo;s global autograd mode. A&#xA;checkpoint can match its recorded hash and still be served by a process that is&#xA;not ready, because &lt;code&gt;model.eval()&lt;/code&gt; does not disable autograd.&lt;/p&gt;&#xA;&lt;p&gt;This example inspects the artefact first, then asks what the process must be true&#xA;of before inference. It uses the cached GPT-2 checkpoint, so the identity fields&#xA;refer to one exact snapshot. The point is not GPT-2 itself; it is the habit of&#xA;recording which bytes and which tensor values the claim belongs to.&lt;/p&gt;</description>
			</item>
			<item>
				<title>Keep a softmax stable when the logits are huge</title>
				<link>http://programmer.ie/examples/inference/03-chapter/</link>
				<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
				<guid>http://programmer.ie/examples/inference/03-chapter/</guid>
				<description>&lt;p&gt;Softmax is usually written as &lt;code&gt;exp(x) / sum(exp(x))&lt;/code&gt;. That formula is correct in&#xA;real arithmetic and unusable in floating point when the logits are large:&#xA;&lt;code&gt;exp(1000)&lt;/code&gt; overflows before the division ever happens. The stable form subtracts&#xA;the maximum logit from every value first. The probabilities are unchanged,&#xA;because the same factor cancels from numerator and denominator.&lt;/p&gt;&#xA;&lt;p&gt;This example runs the stable path on logits &lt;code&gt;1000&lt;/code&gt; and &lt;code&gt;1001&lt;/code&gt;. It also shows the&#xA;selection rules that sit on top of the distribution: top-k keeps the highest&#xA;logits, and top-p keeps the smallest set whose cumulative probability reaches the&#xA;threshold. The example does not use a model, so it isolates the numerical&#xA;mechanism from the quality of any generated text.&lt;/p&gt;</description>
			</item>
			<item>
				<title>Check a transformer&#39;s shapes and causality</title>
				<link>http://programmer.ie/examples/inference/04-chapter/</link>
				<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
				<guid>http://programmer.ie/examples/inference/04-chapter/</guid>
				<description>&lt;p&gt;&amp;ldquo;The transformer is the computation&amp;rdquo; is a claim that can be checked at two&#xA;levels. The first is shape: a batch of five token ids becomes a &lt;code&gt;[1, 5, 8]&lt;/code&gt;&#xA;residual stream, two attention heads produce a &lt;code&gt;[1, 2, 5, 5]&lt;/code&gt; score tensor, and&#xA;the language-model head produces &lt;code&gt;[1, 5, 16]&lt;/code&gt; logits. The second is causality: a&#xA;change to the last input token must leave every earlier position&amp;rsquo;s output&#xA;unchanged.&lt;/p&gt;</description>
			</item>
			<item>
				<title>Watch a generation loop stop on its own</title>
				<link>http://programmer.ie/examples/inference/05-chapter/</link>
				<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
				<guid>http://programmer.ie/examples/inference/05-chapter/</guid>
				<description>&lt;p&gt;An autoregressive loop has two ways to stop: the caller&amp;rsquo;s token limit, or a&#xA;token the model itself chooses, usually an end-of-sequence id. They are not&#xA;interchangeable. A request that stops at &lt;code&gt;max_tokens&lt;/code&gt; may be cut off mid-answer;&#xA;a request that stops on EOS ended where the model intended. A service has to&#xA;report which happened.&lt;/p&gt;&#xA;&lt;p&gt;This example runs the toy decoder from Chapter 04 with an explicit EOS id. The&#xA;interesting part is not the seven generated integers. It is that the loop&#xA;requests eight tokens, emits seven, and returns &lt;code&gt;stop_reason: &amp;quot;eos&amp;quot;&lt;/code&gt; because the&#xA;seventh was token 15. The result object records the per-step token, the time and&#xA;the cumulative length, so the stopping rule is auditable after the fact.&lt;/p&gt;</description>
			</item>
			<item>
				<title>Assemble a request, a backend and a sampler into an engine</title>
				<link>http://programmer.ie/examples/inference/06-chapter/</link>
				<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
				<guid>http://programmer.ie/examples/inference/06-chapter/</guid>
				<description>&lt;p&gt;An inference engine is not a model. It is the code that turns a validated request&#xA;into calls on a model-specific backend, chooses tokens from the returned logits,&#xA;and packages a result with a stop reason and timings. The model can be swapped&#xA;without changing the request contract; the sampler can be swapped without&#xA;changing the backend.&lt;/p&gt;&#xA;&lt;p&gt;This example runs the first engine of the book&amp;rsquo;s Inference Lab. The backend is a&#xA;tiny causal transformer, not GPT-2, and the vocabulary is characters. The point&#xA;is the boundary: &lt;code&gt;Request&lt;/code&gt; carries the decoding policy, &lt;code&gt;ToyBackend&lt;/code&gt; hides the&#xA;model, and &lt;code&gt;InferenceEngine&lt;/code&gt; owns the loop. The output is deterministic for the&#xA;given seed, so a change in one layer can be attributed rather than guessed.&lt;/p&gt;</description>
			</item>
			<item>
				<title>Count the operations before timing the matrix</title>
				<link>http://programmer.ie/examples/inference/07-chapter/</link>
				<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
				<guid>http://programmer.ie/examples/inference/07-chapter/</guid>
				<description>&lt;p&gt;A matrix multiplication of shape &lt;code&gt;(m, n, k)&lt;/code&gt; performs &lt;code&gt;2 * m * n * k&lt;/code&gt;&#xA;floating-point operations, counting one multiply and one add as two. That count&#xA;is fixed by the problem. The time is not: it depends on the device, the layout,&#xA;the precision and the launch overhead. A performance number without the shape and&#xA;the operation count is not interpretable.&lt;/p&gt;&#xA;&lt;p&gt;This example benchmarks one small shape across NumPy on CPU, PyTorch on CPU, and&#xA;PyTorch on CUDA when a GPU is present. On this machine the GPU result was not&#xA;faster, because &lt;code&gt;64 x 128 x 256&lt;/code&gt; is only about 4.2 million FLOPs and the kernel&#xA;launch dominates. That is the lesson, not a GPU ranking. The example also shows&#xA;the warm-up and synchronisation that a timing claim needs.&lt;/p&gt;</description>
			</item>
			<item>
				<title>Compare naive attention with the library kernel</title>
				<link>http://programmer.ie/examples/inference/08-chapter/</link>
				<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
				<guid>http://programmer.ie/examples/inference/08-chapter/</guid>
				<description>&lt;p&gt;Attention writes down a score for every query-key pair. At sequence length &lt;code&gt;T&lt;/code&gt;&#xA;that is &lt;code&gt;T * T&lt;/code&gt; scores per head, before any masking. The arithmetic grows with&#xA;&lt;code&gt;T^2&lt;/code&gt;, even though the number of tokens grows only with &lt;code&gt;T&lt;/code&gt;. That is the cost&#xA;this chapter is about, and it is separate from which kernel computes it.&lt;/p&gt;&#xA;&lt;p&gt;This example compares a hand-written naive attention with PyTorch&amp;rsquo;s scaled&#xA;dot-product attention at 64 positions. The outputs agree to float32 rounding.&#xA;The timings differ on this machine, but the more durable result is the shape of&#xA;the work: two heads, 64 queries, 64 keys, 32 values per head. The page does not&#xA;claim a CUDA speedup; the run used the CPU path and the library may select&#xA;different kernels on another machine.&lt;/p&gt;</description>
			</item>
			<item>
				<title>Separate prefill time from decode time</title>
				<link>http://programmer.ie/examples/inference/09-chapter/</link>
				<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
				<guid>http://programmer.ie/examples/inference/09-chapter/</guid>
				<description>&lt;p&gt;A request&amp;rsquo;s latency is not one number. &lt;strong&gt;Prefill&lt;/strong&gt; processes the prompt in one&#xA;pass and produces the first logits. &lt;strong&gt;Decode&lt;/strong&gt; produces each later token, one&#xA;step at a time. Time to first token (TTFT) includes prefill and the first decode&#xA;step; inter-token latency describes the steps after that. Reporting a single&#xA;average hides which phase dominates and which one a change affected.&lt;/p&gt;&#xA;&lt;p&gt;This example times the two phases separately on the toy transformer. It uses no&#xA;KV cache, so every decode step re-runs the whole sequence. That makes the decode&#xA;times a record of naive recomputation, not a production decode benchmark. The&#xA;value of the example is the accounting: &lt;code&gt;ttft = prefill + first_decode&lt;/code&gt;, and the&#xA;mean inter-token time is the mean of the later decode steps.&lt;/p&gt;</description>
			</item>
			<item>
				<title>Prove a KV cache preserves the answer</title>
				<link>http://programmer.ie/examples/inference/10-chapter/</link>
				<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
				<guid>http://programmer.ie/examples/inference/10-chapter/</guid>
				<description>&lt;p&gt;A KV cache is justified by an identity: for a causal transformer, the keys and&#xA;values at earlier positions do not change when a new token is appended. Reusing&#xA;them should produce the same attention output as recomputing the whole prefix.&#xA;If that identity fails, the cache is a bug, however fast it is.&lt;/p&gt;&#xA;&lt;p&gt;This example checks the identity on random tensors, at a sequence length large&#xA;enough for the loop to do real work. It also exposes a trap. The cached path&#xA;issues one small attention call per token from Python, while the uncached path&#xA;issues one larger batched call. On a CPU, launch and Python overhead can make the&#xA;cached version &lt;em&gt;slower&lt;/em&gt; even though it performs fewer arithmetic operations. The&#xA;example therefore teaches two separate questions: is the answer the same, and is&#xA;the implementation faster?&lt;/p&gt;</description>
			</item>
			<item>
				<title>Place a matmul on the roofline before calling it fast</title>
				<link>http://programmer.ie/examples/inference/11-chapter/</link>
				<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
				<guid>http://programmer.ie/examples/inference/11-chapter/</guid>
				<description>&lt;p&gt;A roofline argument starts with a ratio, not a stopwatch. For a matrix&#xA;multiplication, the operation count is &lt;code&gt;2 * m * n * k&lt;/code&gt; and the bytes moved are&#xA;the three matrices in float32: &lt;code&gt;(m*k + k*n + m*n) * 4&lt;/code&gt;. Their ratio is the&#xA;&lt;strong&gt;arithmetic intensity&lt;/strong&gt; in FLOPs per byte. A machine has a compute ceiling and a&#xA;bandwidth ceiling; the smaller of the two at that intensity bounds the attainable&#xA;rate.&lt;/p&gt;</description>
			</item>
			<item>
				<title>Quantize a matrix and measure what the rounding costs</title>
				<link>http://programmer.ie/examples/inference/12-chapter/</link>
				<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
				<guid>http://programmer.ie/examples/inference/12-chapter/</guid>
				<description>&lt;p&gt;Quantization replaces a wide number format with a narrower one. The saving is&#xA;easy to state: a float32 matrix of 65,536 values occupies 262,144 bytes; the same&#xA;matrix in int8 occupies 65,536 bytes. The cost is also easy to state but harder&#xA;to accept: every value is rounded, and the error depends on the scale.&lt;/p&gt;&#xA;&lt;p&gt;This example uses symmetric per-tensor int8 quantization. One scale is chosen&#xA;from the largest absolute value, every element is divided by that scale, rounded,&#xA;clamped to &lt;code&gt;[-127, 127]&lt;/code&gt;, and later multiplied back. The page reports the maximum&#xA;and mean absolute reconstruction error. It does not claim that the quantized&#xA;model is faster or that its text is equivalent; those are separate experiments.&lt;/p&gt;</description>
			</item>
			<item>
				<title>Find out which attention kernel actually ran</title>
				<link>http://programmer.ie/examples/inference/13-chapter/</link>
				<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
				<guid>http://programmer.ie/examples/inference/13-chapter/</guid>
				<description>&lt;p&gt;FlashAttention is a family of attention implementations that avoid&#xA;materialising the full score matrix in high-bandwidth memory. It is a mechanism,&#xA;not a boolean property of a framework. Asking for it can fail, fall back, or run&#xA;a different kernel, and the result object must say which happened.&lt;/p&gt;&#xA;&lt;p&gt;This example probes three CUDA SDPA backends and a CPU fallback. On this machine&#xA;CUDA was present, but the FlashAttention backend was &lt;strong&gt;not available&lt;/strong&gt;: PyTorch&#xA;returned &lt;code&gt;No available kernel. Aborting execution.&lt;/code&gt; The math and memory-efficient&#xA;backends ran, and the CPU SDPA path ran on the same inputs. The page records that&#xA;absence instead of quoting a speedup from a kernel that did not execute. The&#xA;chapter&amp;rsquo;s separate tiled implementation shows the online-softmax mechanism when&#xA;the library kernel is unavailable.&lt;/p&gt;</description>
			</item>
			<item>
				<title>Allocate paged KV blocks and watch them get reused</title>
				<link>http://programmer.ie/examples/inference/14-chapter/</link>
				<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
				<guid>http://programmer.ie/examples/inference/14-chapter/</guid>
				<description>&lt;p&gt;PagedAttention separates a sequence&amp;rsquo;s &lt;strong&gt;logical&lt;/strong&gt; token positions from the&#xA;&lt;strong&gt;physical&lt;/strong&gt; blocks that hold its keys and values. A sequence of 10 tokens with a&#xA;block size of 4 needs three logical blocks, whether or not the last one is full.&#xA;The allocator hands out physical block ids, and a page table maps one to the&#xA;other. The gain is that sequences no longer need contiguous memory; the cost is&#xA;indirection, partially filled last blocks, and a policy that can still fail.&lt;/p&gt;</description>
			</item>
			<item>
				<title>Step a scheduler through static and continuous batches</title>
				<link>http://programmer.ie/examples/inference/15-chapter/</link>
				<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
				<guid>http://programmer.ie/examples/inference/15-chapter/</guid>
				<description>&lt;p&gt;Static batching waits for a full batch, runs every request to its longest output&#xA;length, and only then admits the next batch. Continuous batching admits a new&#xA;request as soon as a slot frees, at the next model iteration. The difference is&#xA;not a faster matrix multiplication; it is less idle capacity while a short&#xA;request waits behind a long one.&lt;/p&gt;&#xA;&lt;p&gt;This example replays four requests through both policies in integer time steps.&#xA;Static batching produces a makespan of 39 steps and a mean wait of 7.5; continuous&#xA;batching produces 12 steps and 1.25. Those are simulator outputs, not a serving&#xA;benchmark. The value is the schedule: you can point to the step where a slot was&#xA;idle and say why. The simulator&amp;rsquo;s cost model is deliberately simple and is part&#xA;of what the page must disclose.&lt;/p&gt;</description>
			</item>
			<item>
				<title>Make a prefix cache miss on purpose</title>
				<link>http://programmer.ie/examples/inference/16-chapter/</link>
				<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
				<guid>http://programmer.ie/examples/inference/16-chapter/</guid>
				<description>&lt;p&gt;A prefix cache is only safe if its key identifies everything that determines the&#xA;cached value. Two requests with the same token ids but different model weights&#xA;must not share a KV entry. Two requests with the same model but a different token&#xA;prefix must not share one either. The cache must include the scope and the exact&#xA;history, not just a prompt id.&lt;/p&gt;&#xA;&lt;p&gt;This example inserts one prefix under &lt;code&gt;modelA&lt;/code&gt;, then performs three lookups: the&#xA;same prefix under &lt;code&gt;modelA&lt;/code&gt; (hit), the same prefix under &lt;code&gt;modelB&lt;/code&gt; (miss), and a&#xA;prefix that differs in the last token (miss). The output also reports the cache&#xA;size and a deliberately simple hit-rate calculation. The &lt;code&gt;latency_proxy_ms&lt;/code&gt;&#xA;field is a fixed 0.1 versus 1.0, not a measurement; it exists to make the hit/miss&#xA;distinction visible, not to claim a speedup.&lt;/p&gt;</description>
			</item>
			<item>
				<title>Reject every draft token and still match the target</title>
				<link>http://programmer.ie/examples/inference/17-chapter/</link>
				<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
				<guid>http://programmer.ie/examples/inference/17-chapter/</guid>
				<description>&lt;p&gt;Speculative decoding spends extra compute on a cheap draft model to reduce the&#xA;number of target-model passes. The draft proposes several tokens; the target&#xA;checks them. The correctness claim is not that the draft is good. It is that the&#xA;final output matches what the target alone would have produced under the same&#xA;decoding rule.&lt;/p&gt;&#xA;&lt;p&gt;This example runs two cases. In the agreed case, draft and target choose the same&#xA;next token, and five proposals are accepted. In the mismatched case, the draft&#xA;proposes a token the target rejects every time: &lt;code&gt;accepted: 0&lt;/code&gt;, &lt;code&gt;rejected: 6&lt;/code&gt;.&#xA;The generated tokens still equal the baseline. Rejection costs work; it does not&#xA;change the greedy answer. That separation is the mechanism worth learning.&lt;/p&gt;</description>
			</item>
			<item>
				<title>Record a failed compilation instead of hiding it</title>
				<link>http://programmer.ie/examples/inference/18-chapter/</link>
				<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
				<guid>http://programmer.ie/examples/inference/18-chapter/</guid>
				<description>&lt;p&gt;Compilation is an optimisation with a precondition: a compiler and a supported&#xA;graph. When it works, it can remove Python overhead and fuse operations. When it&#xA;fails, the service must either fall back or refuse the request; it must not report&#xA;a compiled path that never ran.&lt;/p&gt;&#xA;&lt;p&gt;This example asks &lt;code&gt;torch.compile&lt;/code&gt; to compile a two-layer MLP. On this machine the&#xA;Inductor backend failed because the C compiler &lt;code&gt;cl&lt;/code&gt; was not found. The result&#xA;object records &lt;code&gt;available: false&lt;/code&gt; and the error, while the eager timings remain&#xA;valid. That is a successful experiment about a failed optimisation. The page does&#xA;not quote a compile speedup, because no compiled invocation completed.&lt;/p&gt;</description>
			</item>
			<item>
				<title>Count the parameters a mixture layer actually uses</title>
				<link>http://programmer.ie/examples/inference/19-chapter/</link>
				<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
				<guid>http://programmer.ie/examples/inference/19-chapter/</guid>
				<description>&lt;p&gt;A mixture-of-experts layer contains many expert networks but evaluates only the&#xA;top-k selected by a router for each token. The model&amp;rsquo;s &lt;strong&gt;total&lt;/strong&gt; parameter count&#xA;includes every expert; the &lt;strong&gt;active&lt;/strong&gt; count for a token includes the router plus&#xA;the selected experts. Those are different numbers, and confusing them turns a&#xA;sparse model into an apparently dense one.&lt;/p&gt;&#xA;&lt;p&gt;This example builds a layer with four experts of dimension 8 and routes a batch&#xA;of four tokens with top-k 2. The router has &lt;code&gt;8 * 4 = 32&lt;/code&gt; weights. Each expert has&#xA;&lt;code&gt;8 * 8 = 64&lt;/code&gt; weights. Active parameters per token are &lt;code&gt;32 + 2 * 64 = 160&lt;/code&gt;; total&#xA;parameters are &lt;code&gt;32 + 4 * 64 = 288&lt;/code&gt;. The run also prints which experts each token&#xA;selected, so the arithmetic can be checked against the routing decisions.&lt;/p&gt;</description>
			</item>
			<item>
				<title>Compare a skewed router with a balanced one</title>
				<link>http://programmer.ie/examples/inference/20-chapter/</link>
				<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
				<guid>http://programmer.ie/examples/inference/20-chapter/</guid>
				<description>&lt;p&gt;A router that sends most tokens to the same expert wastes the other experts and&#xA;overloads one. A capacity limit turns that imbalance into a concrete outcome:&#xA;tokens wait or are dropped. Load variance is one summary, but the maximum load&#xA;and the drop count are the numbers an operator feels.&lt;/p&gt;&#xA;&lt;p&gt;This example routes 100 tokens to four experts with capacity 30 under two&#xA;policies. The skewed policy sends about 70% of tokens to expert 0; the balanced&#xA;policy assigns token &lt;code&gt;t&lt;/code&gt; to expert &lt;code&gt;t % 4&lt;/code&gt;. The skewed run reports a maximum load&#xA;of 60, 36 queued and 6 dropped. The balanced run reports a maximum load of 25,&#xA;no queue and no drops. Those are simulator outputs, and the simulator&amp;rsquo;s capacity&#xA;semantics need reading carefully before quoting them.&lt;/p&gt;</description>
			</item>
			<item>
				<title>Compare KV cache bytes across attention variants</title>
				<link>http://programmer.ie/examples/inference/21-chapter/</link>
				<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
				<guid>http://programmer.ie/examples/inference/21-chapter/</guid>
				<description>&lt;p&gt;The KV cache is often the largest per-request memory cost at long context. Its&#xA;size depends on the number of layers, the sequence length, the number of KV&#xA;heads, the head dimension and the dtype. Multi-head attention stores a key and a&#xA;value for every head. Grouped-query attention stores them for fewer KV heads and&#xA;repeats them at attention time. Multi-query attention stores one KV head.&#xA;Multi-latent attention stores a latent vector per token and projects it back.&lt;/p&gt;</description>
			</item>
			<item>
				<title>Put a token outside the sliding window</title>
				<link>http://programmer.ie/examples/inference/22-chapter/</link>
				<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
				<guid>http://programmer.ie/examples/inference/22-chapter/</guid>
				<description>&lt;p&gt;Sliding-window attention lets a token attend only to itself and the previous&#xA;&lt;code&gt;window - 1&lt;/code&gt; tokens. Positions outside that band are masked before softmax, so&#xA;their weights are exactly zero, not merely small. A stack of such layers extends&#xA;the receptive field, but one layer cannot see beyond the window.&lt;/p&gt;&#xA;&lt;p&gt;This example uses 16 positions and a window of 4. The mask is causal &lt;strong&gt;and&lt;/strong&gt;&#xA;banded: token &lt;code&gt;i&lt;/code&gt; may attend to &lt;code&gt;j&lt;/code&gt; only when &lt;code&gt;j &amp;lt;= i&lt;/code&gt; and &lt;code&gt;i - j &amp;lt; 4&lt;/code&gt;. The&#xA;reported &lt;code&gt;outside_weight_sum&lt;/code&gt; is zero, which is the correctness check. The&#xA;operation counts need care: &lt;code&gt;full_ops&lt;/code&gt; counts the entire &lt;code&gt;16 x 16&lt;/code&gt; matrix,&#xA;including the upper triangle that causal attention already masks, while&#xA;&lt;code&gt;sliding_ops&lt;/code&gt; counts only the band. Comparing those two overstates the saving.&lt;/p&gt;</description>
			</item>
			<item>
				<title>Separate accepting a long prompt from using it</title>
				<link>http://programmer.ie/examples/inference/23-chapter/</link>
				<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
				<guid>http://programmer.ie/examples/inference/23-chapter/</guid>
				<description>&lt;p&gt;A long-context claim bundles at least five separate questions: does the model&#xA;accept the tokens, does it behave correctly at those positions, does the state fit&#xA;in memory, does it answer in acceptable time, and does it use the evidence it was&#xA;given? A benchmark that shows only the first or the fourth has not shown the&#xA;rest.&lt;/p&gt;&#xA;&lt;p&gt;This example runs a toy attention calculation at 1,024, 8,192 and 32,768&#xA;positions. It measures wall-clock time and a Python allocation peak, and it marks&#xA;whether the prompt was truncated at &lt;code&gt;max_context&lt;/code&gt;. It says nothing about a real&#xA;model&amp;rsquo;s quality or retrieval. The value is the separation: the run supports a&#xA;statement about &lt;strong&gt;cost growth&lt;/strong&gt; and &lt;strong&gt;acceptance&lt;/strong&gt;, not about understanding.&lt;/p&gt;</description>
			</item>
			<item>
				<title>Swap runtime adapters under one generation contract</title>
				<link>http://programmer.ie/examples/inference/24-chapter/</link>
				<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
				<guid>http://programmer.ie/examples/inference/24-chapter/</guid>
				<description>&lt;p&gt;A model architecture decides which computations exist: attention shape, KV&#xA;layout, expert routing, positional scheme. A runtime decides how those&#xA;computations are scheduled, batched, cached, compiled and exposed. The two are&#xA;not the same axis, and a service that conflates them ends up claiming a model&#xA;feature because its runtime has a flag for it.&lt;/p&gt;&#xA;&lt;p&gt;This example puts two adapters behind one &lt;code&gt;generate&lt;/code&gt; contract. &lt;code&gt;ToyAdapter&lt;/code&gt;&#xA;reports what it supports: greedy and sampling yes, CUDA and streaming no.&#xA;&lt;code&gt;StrictAdapter&lt;/code&gt; calls &lt;code&gt;require(&amp;quot;streaming&amp;quot;)&lt;/code&gt; and fails with the reason. The&#xA;generation result &lt;code&gt;hixxx&lt;/code&gt; is deliberately trivial; the lesson is the capability&#xA;boundary, not the text. A runtime must ask the adapter what it can do before&#xA;admitting a request that needs it.&lt;/p&gt;</description>
			</item>
			<item>
				<title>Reject an over-budget prompt before generation starts</title>
				<link>http://programmer.ie/examples/inference/25-chapter/</link>
				<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
				<guid>http://programmer.ie/examples/inference/25-chapter/</guid>
				<description>&lt;p&gt;An API that checks a prompt&amp;rsquo;s character length is not checking its token cost.&#xA;The same 1,000 characters can be 221 GPT-2 tokens of English prose, 336 tokens of&#xA;Python source, or 1,000 tokens of spaces. If the context window is 1,024 and the&#xA;request also reserves 256 output tokens, a 4,000-character cap can admit a prompt&#xA;that cannot fit.&lt;/p&gt;&#xA;&lt;p&gt;This example measures those token classes and computes a character cap that is&#xA;safe for the worst class. It then shows the budget function that rejects a&#xA;request before tokenisation or model work. The check is a pure function, so it&#xA;can run in the HTTP layer and in a unit test without loading a model. The page&#xA;does not start a server or call a hosted provider.&lt;/p&gt;</description>
			</item>
			<item>
				<title>Refuse the burst, admit the sustained load</title>
				<link>http://programmer.ie/examples/inference/26-chapter/</link>
				<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
				<guid>http://programmer.ie/examples/inference/26-chapter/</guid>
				<description>&lt;p&gt;An admission controller decides which requests enter the system before any model&#xA;work happens. A queue without a bound converts overload into latency: every&#xA;request is accepted, and every request waits. A bounded queue converts some of&#xA;that overload into immediate refusals, which the client can see and act on.&lt;/p&gt;&#xA;&lt;p&gt;This example sends two arrival patterns through the same controller: ten&#xA;requests at time zero, and ten requests spaced two time units apart. The burst&#xA;accepts 4 and rejects 6; the sustained pattern accepts all 10. The difference is&#xA;not a change in capacity. It is the arrival pattern meeting a fixed queue and&#xA;concurrency limit. The simulator is deliberately small and its queue accounting&#xA;is approximate, so the numbers are a trace, not a capacity plan.&lt;/p&gt;</description>
			</item>
			<item>
				<title>Interleave two streams and cancel one cleanly</title>
				<link>http://programmer.ie/examples/inference/27-chapter/</link>
				<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
				<guid>http://programmer.ie/examples/inference/27-chapter/</guid>
				<description>&lt;p&gt;Streaming is a per-request state machine, not a property of a generator. Each&#xA;request needs its own token queue, its own buffer accounting, and its own&#xA;cancellation state. A manager that mixes them produces interleaved text from two&#xA;users, or keeps generating after the client has gone.&lt;/p&gt;&#xA;&lt;p&gt;This example creates two sessions, takes the first token from each, cancels the&#xA;first, and cleans it up. The output shows that each stream produced its own first&#xA;token, the cancelled session was removed, and one active session remains. The&#xA;buffer limit is two, so the demo also exercises the backpressure check without&#xA;filling it. The page does not claim HTTP streaming or server-sent events; it&#xA;tests the in-process contract that such a transport would use.&lt;/p&gt;</description>
			</item>
			<item>
				<title>Compare four ways to split a model across devices</title>
				<link>http://programmer.ie/examples/inference/28-chapter/</link>
				<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
				<guid>http://programmer.ie/examples/inference/28-chapter/</guid>
				<description>&lt;p&gt;Distributed inference is not one technique. Data parallelism keeps a full model&#xA;on each device and splits requests. Tensor parallelism splits each layer&amp;rsquo;s&#xA;matrices and pays an all-reduce. Pipeline parallelism splits layers into stages&#xA;and pays point-to-point transfers. Expert parallelism routes tokens to different&#xA;experts and pays an all-to-all.&lt;/p&gt;&#xA;&lt;p&gt;This example runs a deterministic cost model for four devices with a 1,000-byte&#xA;model and 100 bytes of activations. Data parallelism reports full compute per&#xA;device and 100 bytes of communication. Tensor parallelism reports one quarter of&#xA;the compute but &lt;code&gt;100 * (4 - 1) = 300&lt;/code&gt; bytes of communication. Pipeline and expert&#xA;parallelism report one quarter of the compute and 100 bytes of communication.&#xA;These are model outputs, not measurements on a multi-GPU machine. The page is&#xA;about the accounting, not a throughput claim.&lt;/p&gt;</description>
			</item>
			<item>
				<title>Price the hand-off in a prefill/decode split</title>
				<link>http://programmer.ie/examples/inference/29-chapter/</link>
				<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
				<guid>http://programmer.ie/examples/inference/29-chapter/</guid>
				<description>&lt;p&gt;Prefill and decode have different resource profiles. Prefill processes many&#xA;tokens at once and is compute-heavy; decode processes one token per request and&#xA;is often memory- and launch-bound. Disaggregation puts them on separate workers&#xA;so each can be sized independently. The price is a hand-off: the KV state must&#xA;move from the prefill worker to the decode worker, and each stage gains a queue.&lt;/p&gt;&#xA;&lt;p&gt;This example runs a hand-built simulator for four requests with prefill cost 5,&#xA;decode cost 1 per token, and transfer cost 2. The co-located schedule completes&#xA;in a total latency of 106; the disaggregated schedule reports 38 with a transfer&#xA;cost of 8. Those numbers are &lt;strong&gt;not&lt;/strong&gt; a measurement of disaggregated hardware.&#xA;The simulator is deliberately simple, and its completion accounting has&#xA;approximations that the page must state.&lt;/p&gt;</description>
			</item>
			<item>
				<title>Cancel, retry and time out one request at a time</title>
				<link>http://programmer.ie/examples/inference/30-chapter/</link>
				<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
				<guid>http://programmer.ie/examples/inference/30-chapter/</guid>
				<description>&lt;p&gt;A serving request can end in more than success. It can be cancelled by the&#xA;client, fail and be retried, exceed a deadline, or exhaust its retry budget.&#xA;Each path has to release resources and report which outcome occurred. &amp;ldquo;The&#xA;process did not crash&amp;rdquo; is not a reliability result.&lt;/p&gt;&#xA;&lt;p&gt;This example runs four controlled cases through a small worker: a clean success,&#xA;an injected failure with one retry, a cancellation after step two, and a timeout.&#xA;Every case releases its request id; the final resource set is empty. The page&#xA;also exposes a boundary defect: with &lt;code&gt;max_retries=0&lt;/code&gt;, an injected failure is not&#xA;raised at all, so the worker reports recovery. That is a useful negative control,&#xA;not a feature.&lt;/p&gt;</description>
			</item>
			<item>
				<title>Parse role and channel boundaries without guessing</title>
				<link>http://programmer.ie/examples/inference/31-chapter/</link>
				<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
				<guid>http://programmer.ie/examples/inference/31-chapter/</guid>
				<description>&lt;p&gt;A protocol is a contract about boundaries: which bytes are a role, which are a&#xA;channel, where content starts, and what happens when a line does not match. A&#xA;model&amp;rsquo;s output is text; the serving layer turns that text into messages. If the&#xA;parser silently drops a malformed line, the next turn runs with missing context.&lt;/p&gt;&#xA;&lt;p&gt;This example parses a four-line sample into &lt;code&gt;system/analysis&lt;/code&gt;, &lt;code&gt;user/final&lt;/code&gt;,&#xA;&lt;code&gt;assistant/analysis&lt;/code&gt; and &lt;code&gt;assistant/final&lt;/code&gt; messages. It uses a deliberately small&#xA;&lt;code&gt;[role|channel] content&lt;/code&gt; grammar, not the full Harmony format. The point is the&#xA;shape of the parse result: a list of messages and a list of errors, both&#xA;preserved. The page does not claim access to OpenAI&amp;rsquo;s serving internals; it&#xA;tests a documented-style message boundary.&lt;/p&gt;</description>
			</item>
			<item>
				<title>Evaluate a cache policy without claiming provider internals</title>
				<link>http://programmer.ie/examples/inference/32-chapter/</link>
				<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
				<guid>http://programmer.ie/examples/inference/32-chapter/</guid>
				<description>&lt;p&gt;Provider prompt caching is an API-level optimisation with a contract: a minimum&#xA;cacheable prefix, a write cost, a read cost, and an expiry. The contract is&#xA;documented by the provider, but the internal cache implementation is not. A&#xA;provider-neutral experiment can model the contract and its break-even; it cannot&#xA;claim what the provider&amp;rsquo;s servers did.&lt;/p&gt;&#xA;&lt;p&gt;This example runs a mock eligibility rule. A message with 1,500 tokens and a&#xA;&lt;code&gt;cache_control&lt;/code&gt; block is eligible because the total is at least 1,024 tokens. The&#xA;result reports 1,500 cache-creation tokens, zero cache-read tokens and an&#xA;&lt;code&gt;estimated_savings&lt;/code&gt; of &lt;code&gt;0.9&lt;/code&gt;. That 0.9 is a placeholder in the source, not a&#xA;measured discount. The page uses the example to teach the policy boundary, and&#xA;explicitly refuses to turn the placeholder into a provider claim.&lt;/p&gt;</description>
			</item>
			<item>
				<title>Spend a token budget and watch accuracy saturate</title>
				<link>http://programmer.ie/examples/inference/33-chapter/</link>
				<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
				<guid>http://programmer.ie/examples/inference/33-chapter/</guid>
				<description>&lt;p&gt;A reasoning model can spend more tokens before its final answer. That extra&#xA;compute may improve accuracy on hard tasks, but it is not free: latency and cost&#xA;grow with the budget. A useful experiment reports the accuracy curve and the&#xA;cost curve together, and marks where additional budget stops helping.&lt;/p&gt;&#xA;&lt;p&gt;This example uses a synthetic solver: a task is solved when the budget is at&#xA;least the task string&amp;rsquo;s length, latency is the budget, and cost is the budget&#xA;times &lt;code&gt;0.001&lt;/code&gt;. It is not a reasoning model and not a provider measurement. It is&#xA;an accounting fixture for the trade-off. With budgets 2, 5 and 10, accuracy is&#xA;&lt;code&gt;5/9&lt;/code&gt; and total cost proxy is &lt;code&gt;0.051&lt;/code&gt;; the hard task needs 10, while the easy&#xA;tasks saturate earlier.&lt;/p&gt;</description>
			</item>
			<item>
				<title>Refuse a tool call that cannot be executed</title>
				<link>http://programmer.ie/examples/inference/34-chapter/</link>
				<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
				<guid>http://programmer.ie/examples/inference/34-chapter/</guid>
				<description>&lt;p&gt;A tool call is a request to execute code with model-supplied arguments. The&#xA;serving layer must decide whether the tool exists, whether the arguments are&#xA;valid, what the tool may touch, and how to report a refusal. A loop that catches&#xA;every exception and retries forever turns a bad call into latency; a loop that&#xA;passes the argument to &lt;code&gt;eval&lt;/code&gt; turns it into a security incident.&lt;/p&gt;&#xA;&lt;p&gt;This example runs a mocked calculator loop with two valid expressions and one&#xA;unknown tool. The valid calls return &lt;code&gt;4&lt;/code&gt; and &lt;code&gt;42&lt;/code&gt;; the unknown tool returns&#xA;&lt;code&gt;error&lt;/code&gt; after three attempts. The implementation uses Python&amp;rsquo;s &lt;code&gt;eval&lt;/code&gt; with&#xA;builtins removed, which is &lt;strong&gt;not&lt;/strong&gt; a safe sandbox. The repository&amp;rsquo;s&#xA;&lt;code&gt;tool_safe.py&lt;/code&gt; is the alternative: it parses an AST and permits only numbers and&#xA;six arithmetic operators, with length, node, integer and exponent bounds.&lt;/p&gt;</description>
			</item>
			<item>
				<title>Separate validating JSON from constraining decoding</title>
				<link>http://programmer.ie/examples/inference/35-chapter/</link>
				<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
				<guid>http://programmer.ie/examples/inference/35-chapter/</guid>
				<description>&lt;p&gt;There are two different guarantees hiding under &amp;ldquo;structured output&amp;rdquo;. A validator&#xA;checks text &lt;strong&gt;after&lt;/strong&gt; the model has produced it and may reject or repair it. A&#xA;constrained decoder changes the next-token distribution &lt;strong&gt;during&lt;/strong&gt; generation so&#xA;that invalid tokens cannot be chosen. The first can fail after paying for the&#xA;whole generation; the second needs a schema compiler and a token mask.&lt;/p&gt;&#xA;&lt;p&gt;This example demonstrates both primitives at a small scale. &lt;code&gt;validate_json&lt;/code&gt;&#xA;accepts a complete record, rejects a record missing &lt;code&gt;value&lt;/code&gt;, and rejects malformed&#xA;JSON. &lt;code&gt;FiniteStateDecoder&lt;/code&gt; filters a candidate list down to allowed tokens. The&#xA;filter is a mask primitive, not a model loop: the page does not claim that a model&#xA;was constrained. The chapter&amp;rsquo;s &lt;code&gt;structured_lab.py&lt;/code&gt; is where the schema compiler is&#xA;wired into a real model, and it is optional here.&lt;/p&gt;</description>
			</item>
			<item>
				<title>Compute TTFT, TPOT and throughput from a request trace</title>
				<link>http://programmer.ie/examples/inference/36-chapter/</link>
				<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
				<guid>http://programmer.ie/examples/inference/36-chapter/</guid>
				<description>&lt;p&gt;Latency reports are easy to make incomparable. Time to first token (TTFT) is not&#xA;inter-token latency; time per output token (TPOT) is not end-to-end latency; and&#xA;throughput needs a denominator. Two services can report the same &amp;ldquo;tokens per&#xA;second&amp;rdquo; while measuring different intervals, token counts or concurrency.&lt;/p&gt;&#xA;&lt;p&gt;This example aggregates two request records. It computes TTFT p50 and p99, TPOT&#xA;p50, end-to-end p50, and a token rate from total output tokens over the sum of&#xA;per-request end-to-end times. The numbers are small and deterministic. The&#xA;important part is the definition: the throughput denominator here is &lt;strong&gt;additive&#xA;request time&lt;/strong&gt;, not wall-clock time under concurrency. It is a per-trace rate, not&#xA;a serving capacity.&lt;/p&gt;</description>
			</item>
			<item>
				<title>Recompute cost per million tokens from pinned assumptions</title>
				<link>http://programmer.ie/examples/inference/37-chapter/</link>
				<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
				<guid>http://programmer.ie/examples/inference/37-chapter/</guid>
				<description>&lt;p&gt;Cost per million tokens is arithmetic, but only after four assumptions are&#xA;explicit: how many tokens are counted, what the hardware costs per hour, how many&#xA;tokens per second it produces, and what fraction of the hour is actually used.&#xA;Change any one and the result moves. A cost table without those inputs is a&#xA;price, not an estimate.&lt;/p&gt;&#xA;&lt;p&gt;This example uses one million tokens, hardware at &lt;code&gt;$2.00&lt;/code&gt; per hour, a throughput&#xA;of 100 tokens per second and 50% utilisation. The effective rate is 50 tokens per&#xA;second, so one million tokens take 20,000 seconds, or 5.555 hours, and cost&#xA;&lt;code&gt;$11.11&lt;/code&gt;. The sensitivity sweep at 25%, 50%, 75% and 100% utilisation gives&#xA;&lt;code&gt;$22.22&lt;/code&gt;, &lt;code&gt;$11.11&lt;/code&gt;, &lt;code&gt;$7.41&lt;/code&gt; and &lt;code&gt;$5.56&lt;/code&gt;. These are model outputs from stated&#xA;assumptions, not a bill from a provider.&lt;/p&gt;</description>
			</item>
			<item>
				<title>Catch a capacity model that contradicts its own data</title>
				<link>http://programmer.ie/examples/inference/38-chapter/</link>
				<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
				<guid>http://programmer.ie/examples/inference/38-chapter/</guid>
				<description>&lt;p&gt;A capacity model is a claim about how throughput responds to load. It should be&#xA;fitted to observations, checked against points it did not use, and refused outside&#xA;the observed range. A model that predicts 16 tokens per second at load 4 when the&#xA;measurement at load 4 was 30 has failed its own data, even if its API is tidy.&lt;/p&gt;&#xA;&lt;p&gt;This example builds a four-point capacity table: loads 1, 2, 4 and 8 with&#xA;throughputs 10, 18, 30 and 32. It then predicts throughput at load 4 and rejects&#xA;a prediction at load 100. The prediction at 4 is 16, because the model scales&#xA;linearly from the maximum observed load rather than fitting the points. The&#xA;extrapolation guard is correct; the interpolation is not. The page uses the&#xA;contradiction as the lesson: a guard against extrapolation is not a substitute&#xA;for a fit.&lt;/p&gt;</description>
			</item>
			<item>
				<title>Replay a computation exactly, then break batch invariance</title>
				<link>http://programmer.ie/examples/inference/39-chapter/</link>
				<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
				<guid>http://programmer.ie/examples/inference/39-chapter/</guid>
				<description>&lt;p&gt;Reproducibility has levels. The same program with the same seed on the same&#xA;device may replay bit-for-bit. The same computation with a different batch shape&#xA;may not. A test that only checks the first property can pass while a serving&#xA;system silently changes its answers when requests are batched together.&lt;/p&gt;&#xA;&lt;p&gt;This example runs two checks. &lt;code&gt;run_twice&lt;/code&gt; seeds PyTorch, executes a function,&#xA;reseeds, executes again, and compares with &lt;code&gt;torch.equal&lt;/code&gt;. &lt;code&gt;batch_invariance&lt;/code&gt;&#xA;computes the first sample&amp;rsquo;s output at batch 1 and batch 4 and compares them. The&#xA;replay is exactly equal; the batch comparison differs by &lt;code&gt;5.378&lt;/code&gt;, so&#xA;&lt;code&gt;invariant: false&lt;/code&gt;. The second result is the important one: a batch-size change is&#xA;a numerical change, not just a scheduling change.&lt;/p&gt;</description>
			</item>
			<item>
				<title>Expose the metric trap in an averaged benchmark</title>
				<link>http://programmer.ie/examples/inference/40-chapter/</link>
				<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
				<guid>http://programmer.ie/examples/inference/40-chapter/</guid>
				<description>&lt;p&gt;A benchmark headline can be mathematically correct and still mislead. An average&#xA;over two workloads answers &amp;ldquo;what is the mean of these trials?&amp;rdquo; It does not answer&#xA;&amp;ldquo;how fast is a short request?&amp;rdquo;, &amp;ldquo;how fast is a long request?&amp;rdquo;, or &amp;ldquo;what happens&#xA;at this arrival rate?&amp;rdquo;. If the workloads have different sizes and the denominator&#xA;is not stated, the mean is not a service-level result.&lt;/p&gt;&#xA;&lt;p&gt;This example builds six trials from two workloads. The &amp;ldquo;short prompt&amp;rdquo; has 12&#xA;characters; the longer prompt has 32. Latency is &lt;code&gt;0.1 + 0.01 * length&lt;/code&gt;, so the&#xA;short trials are 0.22 and the long ones 0.42. The mean is 0.32, and the 95%&#xA;interval is about &lt;code&gt;[0.232, 0.408]&lt;/code&gt;. The interval spans both workloads because the&#xA;trials are pooled. The &lt;code&gt;tokens&lt;/code&gt; field is the string length, not a model token&#xA;count, and the three repeats are deterministic copies, not independent samples.&lt;/p&gt;</description>
			</item>
			<item>
				<title>Write the runtime contract before the implementation</title>
				<link>http://programmer.ie/examples/inference/41-chapter/</link>
				<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
				<guid>http://programmer.ie/examples/inference/41-chapter/</guid>
				<description>&lt;p&gt;A runtime design is a set of decisions about boundaries: what a request contains,&#xA;what a backend must provide, what the engine returns, and which dependencies may&#xA;cross the boundary. Writing those decisions down before optimising makes it&#xA;possible to test a backend against a contract rather than against a particular&#xA;implementation.&lt;/p&gt;&#xA;&lt;p&gt;This example prints two architecture decision records and one &lt;code&gt;RuntimeContract&lt;/code&gt;,&#xA;then runs a toy backend through &lt;code&gt;validate_contract&lt;/code&gt;. The contract names&#xA;&lt;code&gt;InferenceEngine&lt;/code&gt;, inputs &lt;code&gt;prompt&lt;/code&gt; and &lt;code&gt;max_new_tokens&lt;/code&gt;, and outputs &lt;code&gt;text&lt;/code&gt; and&#xA;&lt;code&gt;tokens&lt;/code&gt;. The check asks the backend for &lt;code&gt;hi&lt;/code&gt; plus two tokens and expects &lt;code&gt;hixx&lt;/code&gt;.&#xA;It is a smoke test, not a general type checker, and the page says so.&lt;/p&gt;</description>
			</item>
			<item>
				<title>Wire a scheduler and cache into one admission path</title>
				<link>http://programmer.ie/examples/inference/42-chapter/</link>
				<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
				<guid>http://programmer.ie/examples/inference/42-chapter/</guid>
				<description>&lt;p&gt;Scheduling and caching are separate mechanisms, but they meet at admission. A&#xA;request may be rejected because the queue is full, or admitted and then find a&#xA;cached prefix. The order matters: checking the cache before admission can waste&#xA;work on a request that will be refused; admitting before checking can reserve a&#xA;slot for a request that is already complete.&lt;/p&gt;&#xA;&lt;p&gt;This example runs a miniature integration. A scheduler with a queue limit of four&#xA;receives five requests, so four are scheduled and one is refused. A cache is&#xA;consulted for each scheduled request id. Because every id is new, there are zero&#xA;hits. One scheduled request is completed and cleaned. The output is a wiring&#xA;check, not a hit-rate or throughput result.&lt;/p&gt;</description>
			</item>
			<item>
				<title>Run one conformance check against three backend names</title>
				<link>http://programmer.ie/examples/inference/43-chapter/</link>
				<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
				<guid>http://programmer.ie/examples/inference/43-chapter/</guid>
				<description>&lt;p&gt;A multi-backend runtime needs one request contract and several adapters. The&#xA;contract can be checked with a conformance suite; the suite can tell you that an&#xA;adapter is callable and returns the expected shape. It cannot tell you that two&#xA;models produce the same answers, or that an unavailable engine would pass if it&#xA;were installed.&lt;/p&gt;&#xA;&lt;p&gt;This example runs the same check against three names. &lt;code&gt;pytorch-toy&lt;/code&gt; is available&#xA;and passes because it returns &lt;code&gt;hixx&lt;/code&gt; for &lt;code&gt;hi&lt;/code&gt; plus two tokens. &lt;code&gt;vllm&lt;/code&gt; and &lt;code&gt;sglang&lt;/code&gt;&#xA;are unavailable stubs, so they are recorded as &lt;code&gt;available: false&lt;/code&gt;, &lt;code&gt;passed: false&lt;/code&gt; with a reason. The output is an eligibility matrix, not a benchmark. The&#xA;check is deliberately weak: a constant function could pass it.&lt;/p&gt;</description>
			</item>
			<item>
				<title>Separate a benchmark plan from a measured comparison</title>
				<link>http://programmer.ie/examples/inference/44-chapter/</link>
				<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
				<guid>http://programmer.ie/examples/inference/44-chapter/</guid>
				<description>&lt;p&gt;A benchmark has two artefacts that are easy to confuse: a &lt;strong&gt;plan&lt;/strong&gt; describing&#xA;what would be compared, and a &lt;strong&gt;result&lt;/strong&gt; describing what actually ran. The plan&#xA;is cheap and can list any engine. The result requires an installed engine, a&#xA;pinned version, a frozen workload, a measurement point and records that another&#xA;person can inspect. Reporting the first as the second is the central failure&#xA;this chapter is written to prevent.&lt;/p&gt;</description>
			</item>
			<item>
				<title>Audit the book&#39;s evidence without manufacturing completion</title>
				<link>http://programmer.ie/examples/inference/45-chapter/</link>
				<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
				<guid>http://programmer.ie/examples/inference/45-chapter/</guid>
				<description>&lt;p&gt;The final chapter makes a claim about the book&amp;rsquo;s own evidence. Every chapter has&#xA;a manuscript, at least one test file and a result file. That is an inventory.&#xA;It is not proof that the experiments ran, that the results are correct, or that&#xA;an independent person reproduced them. The audit has to say so in fields, not in&#xA;a hopeful sentence.&lt;/p&gt;&#xA;&lt;p&gt;This example walks the repository and builds a 45-entry manifest. Each entry&#xA;records whether the manuscript, tests and result exist, the result&amp;rsquo;s declared&#xA;status, and a SHA-256 for every artefact. The audit then reports&#xA;&lt;code&gt;inventory_complete: true&lt;/code&gt; because all 45 entries are present, and&#xA;&lt;code&gt;all_complete: false&lt;/code&gt;, &lt;code&gt;tests_executed_by_audit: false&lt;/code&gt;,&#xA;&lt;code&gt;claims_verified_by_audit: false&lt;/code&gt; and &lt;code&gt;independent_review: NOT_RUN&lt;/code&gt; because the&#xA;inventory does not do those things. That combination is the honest conclusion.&lt;/p&gt;</description>
			</item>
	</channel>
</rss>
