Reuse Before Recompute

A team running a long debugging session finally acts on Chapter 4’s advice. Their traces show 20,000 tokens of duplicate file reads and obsolete tool output, so they add a cleanup pass that strips the duplicates before every request. Token counts fall by a sixth. The next invoice rises. Latency does not improve either. Nothing about the model changed and nothing about the task changed. The cleanup was real, the savings were arithmetic, and the bill disagrees, because the bill never measured tokens. It measured computation, and the cleanup destroyed the reuse that had been quietly subsidising every turn.

Chapter 8 ended with recomputation waste: identical prefixes processed from scratch each request because no interface remembers prior work. This chapter is about the interfaces that do remember, and about the layout discipline they demand. Prompt-prefix caching is usually presented as a billing feature. For this book it is something more consequential: the first mechanism where the shape of the context, not just its size, determines what the session costs.

Two caches, held apart

Chapter 8 established the firewall and this chapter patrols it, because every paragraph below dies if the two caches merge. The inference KV cache, or KV state, is model-side internals: keys and values retained during a forward pass so generation need not recompute the past. It lives inside one request’s computation and is rebuilt whenever the runtime sends a new request. The prompt-prefix cache, also called the prompt cache or provider prefix cache, is a serving-layer mechanism: when a new request’s rendered leading tokens match a previously computed prefix, the provider reuses that work instead of redoing it. The two are related, OpenAI’s documentation describes its prompt cache directly as preserved KV tensors, but they are not the same engineering object. Different owners, different granularity, different controls. This chapter says “prompt cache” for the serving concept and “KV state” for the model concept, always, and treats any sentence where either reading is possible as a defective sentence.

The primitive: repeated context need not mean repeated computation

An agent session is a study in repetition. A hundred turns may all begin with the same system instructions, the same tool definitions, the same project rules, the same early history, diverging only in the latest observations and the current request. The naive computational model processes everything every time:

request 1    process A B C D
request 2    process A B C D E
request 3    process A B C D E F

With reusable prefix computation, the shared work happens once:

request 1    compute A B C D
request 2    reuse A B C D, compute E
request 3    reuse A B C D E, compute F

The chapter’s primitive follows: repeated context does not necessarily imply repeated computation, but only while the reusable part stays structurally compatible with what the cache holds. Compatibility is the entire subject. Everything below describes what preserves it, what breaks it, and what it costs to create it in the first place.

The geometry of a reusable prefix

Take two requests, R1 = A B C D E and R2 = A B C D F. Their reusable prefix is A B C D. At the first divergence, E against F, reuse stops, because nothing about the matched prefix certifies the unmatched suffix. The geometry generalises:

shared prefix
        ↓
candidate reusable computation

first divergence
        ↓
new computation required

Two qualifications keep this honest across providers. First, no provider promises literal token-by-token global matching as its implementation; each documents its own units of reuse, breakpoints, prefix units, lookup windows, and the chapter treats those as separate mechanisms sharing one geometry rather than one mechanism with different labels. Second, reuse is attempted, never guaranteed: routing, expiry, and best-effort service mean an identical prefix can still miss, a fact the lifetime section makes operational. The geometry describes what can be reused. Provider mechanics describe what is.

Open serving systems show the same geometry without any commercial API attached. SGLang’s RadixAttention organises prompts in a radix tree so that requests sharing prefixes share KV-state computation, which is exactly the candidate-reuse shape above implemented as scheduler data structure. The book cites it as the open counterpart that proves the principle is architectural rather than vendor-specific, and draws no inference whatsoever about what any commercial provider runs internally.

Stable first, volatile later, within semantic law

Chapter 6 established stable-versus-dynamic placement as an ordering distinction and left the economic payoff uncollected. This chapter collects it. A cache-friendly bundle has a characteristic shape:

STABLE
system and product instructions
tool definitions
project rules
stable shared reference material

        ↓

SEMI-STABLE
conversation history
previous tool calls and results

        ↓

DYNAMIC
current state
current observation
current user message

Stable material leads because every request reuses it; volatile material trails because each request recomputes from the divergence anyway. The providers’ own guidance converges on this shape from different directions. OpenAI’s documentation advises placing stable developer instructions and shared reference material first, pushing timestamps and user-specific content late or into later messages. Anthropic’s breakpoint guidance says to mark the last block whose prefix stays identical across the requests meant to share a cache. Kimi’s documentation is blunter still: fixed content first, conversation appended at the end, or the cache cannot be hit. Three vendors, one geometry.

The shape is a hypothesis to test, not a licence to reorder arbitrarily, because authority and semantics constrain layout first. Moving a project rule after the observations it governs may improve prefix stability while breaking instruction precedence; burying the current request early to protect a history cache may satisfy the cache and starve the task. Chapter 6’s canonical layout already encodes authority-before-data and framing-before-evidence, and cache placement optimises strictly within the layouts those constraints leave legal. The reconciliation, recorded here as policy:

1. correctness and authority requirements
2. behavioural ordering requirements
3. freshness requirements
4. cache-friendly placement where still free to choose

Caching never outranks correctness. A layout that buys hits with misplaced governance is not optimised. It is corrupt.

Append versus rewrite

The largest structural choice a harness makes is whether history grows by addition or by revision, and caching prices the two oppositely. Append-only growth preserves every earlier prefix:

turn 1    A B
turn 2    A B C
turn 3    A B C D
turn 4    A B C D E

Each turn’s request extends the last, so the reusable prefix grows monotonically and every earlier computation stays valid. Historical rewrite does the opposite:

turn 1    A B
turn 2    A B C
turn 3    A SUMMARY(B C)
turn 4    A SUMMARY2(B C D) E

The rewritten request is shorter at every step past the first edit, and every rewrite moves the divergence point backward, orphaning the cached computation of everything after it. This is the economic conflict Chapters 10 through 12 inherit and must respect: deletion and summarisation buy token reduction with cache invalidation, and neither side of the trade is visible from token counts alone. The chapter does not resolve the conflict. It prices it, and the price arrives in the next section.

Mutation radius

A small edit early can cause a large recomputation footprint, and the concept deserves a name because token-diff thinking misses it completely. Call it mutation radius: how much downstream prefix reuse is lost when an earlier part of the rendered context changes. The term earns its place the first time a reader sees a one-line timestamp edit invalidate 80,000 tokens of cached prefix, an event every provider’s documentation describes in its own vocabulary. OpenAI’s gotcha pages show extending a message, switching breakpoint modes, or rewriting developer content orphaning everything after the change. Anthropic’s timestamp-trap example shows a per-request block defeating an entire static prefix because no write ever accumulated behind it. Kimi’s tool-loading guidance warns that mid-conversation insertion invalidates from the change point onward while append-only loading preserves the established prefix.

The useful content of the term is one asymmetry:

small edit size
        ≠
small recomputation footprint

positioned early, the first dominates the second by orders of magnitude. Every later chapter that mutates history, and all of them do, must report its mutation radius alongside its token savings, or its savings are unaudited. That reporting requirement is Chapter 9’s main bequest to Chapter 10.

Writes, reads, lifetimes: the amortisation curve

A reusable prefix is not free to create, and the economics turn on who pays for creation. Conceptually every provider implements some version of the same curve:

total cost
  = initial write or computation
  + Σ later cache reads
  + Σ uncached suffix computation

against

total uncached cost
  = Σ full-request computation

A cacheable prefix repays its setup cost only through enough reuse before expiry, which makes caching an amortisation problem rather than a discount. The current OpenAI semantics for GPT-5.6 and later models, verified September 2026, make the cleanest worked example the book has yet had. Cache writes cost 1.25 times the ordinary input rate; subsequent reads cost 0.1 times. Writing a prefix once and fully reusing it once costs 1.35 times its ordinary input cost against 2 times without caching; across ten requests, one write plus nine full reads costs 2.15 times against 10 times. The break-even is nearly immediate for genuinely repeated prefixes and never arrives for one-shot content, which is why explicit-only mode exists: content unlikely to be reused should never incur the write charge at all.

Anthropic’s surface differs in controls while rhyming in economics: writes at 1.25 times base for the default five-minute lifetime or twice base for the one-hour option, reads at a tenth with per-model exceptions, a maximum of four breakpoints, and telemetry that separates created, read, and post-breakpoint tokens. DeepSeek’s disk cache differs again: no separately documented write charge, automatic prefix-unit persistence with full-match rules, hit and miss tokens reported per request, best-effort retention over hours to days. Kimi removes the controls entirely: fully automatic caching past a 256-token floor, no TTL to manage, discounted hit billing, with layout advice as the only lever. Four implementations, one curve: pay to create, discount to reuse, expire eventually.

Lifetime is the curve’s third term and the easiest to forget. Anthropic measures entry life from the start of the writing or reading request, with free refresh on reuse, so a four-minute response leaves roughly one minute for the follow-up under the default TTL. OpenAI’s GPT-5.6 generation offers a thirty-minute minimum TTL from latest write or reuse. DeepSeek promises nothing beyond best effort over hours to days. A miss therefore proves nothing about layout on its own: identical prefixes miss across expiry, across routing changes, across regions, across model or configuration changes. The experiment below records intervals and settings for exactly this reason. Structurally reusable and currently available are different claims, and only the second decides the bill.

The minimum-length trap

Some providers decline to cache short prefixes at all, which breaks the naive rule that fewer tokens cost less. OpenAI’s GPT-5.6 floor is 1,024 tokens; Anthropic’s minima run from 512 to 4,096 by model; Kimi’s automatic cache ignores prior requests under 256 tokens. Below the floor, a stable prefix earns zero reuse however perfectly arranged. Just above it, the same material amortises across the session. The documentation works the resulting break-even explicitly: with a 1,024-token floor, tenth-rate reads, and 1.25-rate writes, a prefix needs on the order of a hundred-plus tokens of stable content with realistic reuse before expansion beats brevity, and the tiniest prefixes never benefit at any request count.

The chapter recommends nothing padded. Useless text injected to cross a threshold buys cache eligibility with interference, the exact trade Chapter 5 forbids. Where the docs advise expansion, they mean useful stable material, examples, reference content, calibration text, or else shortening what cannot earn reuse. The trap matters here as pure first principles: it is the cleanest case in the book where the token-minimal bundle is not the cost-minimal bundle, stated with numbers rather than slogans.

Breakpoints are context decisions

Where providers expose explicit breakpoints, caching policy becomes context management by another name. Each breakpoint declares that the prefix behind it is stable enough to pay for saving, and each omission declares the suffix too volatile to bother writing. The decision variables are now visible: expected reuse count, prefix size, volatility, lifetime, write cost, read discount, latency sensitivity. OpenAI’s explicit mode makes the economics literal, up to four writes per request, unmarked suffixes processed uncached with no write charge. Anthropic’s model makes the failure mode literal instead: writes happen only at breakpoints, reads search backward through a twenty-block window for prior writes, so a breakpoint placed on per-request content writes expensively every turn and reads never. The general lesson needs no vendor: breakpoint placement is admission policy for computation, and it belongs beside the admission policy for content.

Tools and rules as cache examples

Two standing costs from earlier chapters return with second properties. Tool definitions, Chapter 3’s standing tax, are often the most reusable bytes in the bundle: forty schemas unchanged across hundreds of requests form a prefix worth writing once and reading forever. Mutate one schema per request, a timestamp, a dynamic identifier, session-specific text, early enough in the order, and the mutation radius wipes out everything downstream. OpenAI’s guidance is explicit because the failure is common: keep definitions, ordering, and schemas stable, disable tools per request by choice rather than by removal, defer loading so discovered tools append late instead of churning early. None of this is Chapter 17’s tool design. It is the cache ledger’s view of the same objects.

Project instruction files behave identically. AGENTS.md and CLAUDE.md change rarely, which makes them excellent prefix material, until someone embeds the current time, a session identifier, or live branch status inside the early standing block. Chapter 3’s hidden context becomes economically observable here: bytes nobody typed and nobody reviews, invalidating reuse on every turn through pure volatility. The fix costs nothing behavioural, render dynamic values late or not at all, and it is invisible without the telemetry this chapter’s experiment records.

Freshness, ordering, retention: three reconciliations

Cache stability collides with three earlier constraints, and each collision resolves the same way: correctness first, cache second.

Freshness first. A stable stale snapshot, a file state the repository has moved past, reuses beautifully and answers wrongly. Cache stability is not information validity, and no hit rate justifies retaining what is incorrect. Chapter 20 will own validity machinery; this chapter establishes only the precedence, because a compiler optimising for hits will otherwise learn to prefer the stale. Correctness dominates cache optimisation, and any future objective function must encode that ordering, not discover it.

Ordering second. Chapter 6 showed that sequence changes behaviour through authority, salience, and adjacency, none of which consult the cache. The reconciliation from the stable-first section stands: semantic and authority constraints define the legal layouts, and cache stability chooses among them. “Put all stable tokens first” as an unconditional rule would happily demote a governing instruction beneath volatile observations. The legal-layouts framing is what lets Chapters 6 and 9 agree instead of competing.

Retention third. Chapter 7’s classes describe what information requires, not what reuses well, and the two dimensions cross freely. PIN plus highly volatile is important and poorly reusable: a live deployment lock consulted every turn but rewritten every turn. REFETCHABLE plus stable is eminently cacheable: project reference material re-read rarely but reused constantly while present. Retention semantics and cacheability are independent axes, and the chapter introduces no new taxonomy field for the second because the existing telemetry already measures it: realised read ratios per span are cacheability observed, no label required. If Context Lab ever needs a cacheability predictor rather than a measurement, that field must earn its place against observed ratios. It has not yet.

Cacheability is an economic property, not a relevance judgement

That sentence is worth isolating because the whole chapter compresses into it. Nothing about a span’s hit rate says anything about whether the span should be there. A cached irrelevance is still an irrelevance; it merely costs less to be irrelevant. Conversely, an uncacheable necessity, the genuinely novel observation each turn must carry, is not waste because it misses. Teams that optimise the hit-rate dashboard instead of the bundle will keep cheap filler and starve live signal, repeating at the economic level the exact error Chapter 5 diagnosed at the behavioural level. The instrument reports reuse. Judgement about membership stays with content, retention, and ordering policy.

What caching cannot solve

The boundary needs stating without softening, because caching success feels like context success. Prefix reuse reduces recomputation, latency, and input-processing cost. It does not touch window occupancy: cached tokens still count against capacity on the providers that count them, and the bundle still fills. It does not touch interference, position sensitivity, staleness, authority conflict, or bloat. A million cached irrelevant tokens could still be bad context even if processing them were free. Hence the chapter’s cleanest sentence, kept verbatim:

Caching can make bad context cheaper.

That is why caching and selection are complements rather than substitutes, and why the runtime chapters survive every cache improvement. Selection decides what deserves to be there. Caching decides what repeated presence costs. Neither answers the other’s question.

Proposed experiment: reuse economics

The design measures money and computation, not intelligence, so the task is deliberately trivial and deterministic: fixed lookups with checkable answers, where model quality cannot dominate. The primary provider is OpenAI on a GPT-5.6-generation model, chosen because its telemetry separates ordinary, write, and cached tokens per request. A secondary reduced run on DeepSeek tests whether the prefix geometry transfers across a disk-backed automatic cache with hit/miss reporting. Anthropic enters as documentation case for breakpoint and TTL policy surfaces, not as a third full implementation.

Construct one large stable prefix: system and project instructions, tool definitions, reference material, frozen prior history, all byte-frozen with versions recorded. Append a small dynamic suffix carrying the per-turn task. Then run five conditions with identical semantics throughout, using synthetic inert fields for mutations so behaviour cannot confound cost:

A  stable append-only:
   PREFIX + turn1, PREFIX + turn1 + turn2, ...

B  early-prefix mutation:
   same prefix with a small inert change near the beginning

C  late-suffix mutation:
   same-sized inert change after the main cacheable prefix

D  historical rewrite:
   an older segment replaced by a fixed-length synthetic summary,
   mechanically constructed, never LLM-generated

E  no-reuse control:
   enough prefix churn to guarantee cold computation

Condition D is a cache-invalidation probe, not a compression algorithm: its summary is synthetic and fixed-length precisely so that Chapter 11’s behavioural questions stay out of this measurement. Condition E calibrates the ceiling every other condition is measured against.

Record per request: total visible input tokens, cache-write tokens, cache-read tokens, cache-miss tokens where exposed, uncached tokens, time to first token, total latency, provider-reported cost inputs with independently calculated API cost, prefix size, first-divergence position, request interval, and cache TTL configuration. These are invocation-level economic observations, recorded against the request, not metadata on every context item: structure and economics stay in distinct records that join on the bundle. Where a provider hides a metric, record the absence rather than estimating it. The pre-registered readings: B should show large mutation radius against small edit size; C should show localised cost; D should show the rewrite penalty Chapter 10 must beat; A against E bounds the session value of reuse. The timing component repeats the core A-loop at immediate, short-delay, near-TTL-boundary, and post-TTL intervals, confirming that identical prefixes expire rather than persist. If rate limits or cost make the timing arm impractical, specify it unrun rather than claiming it.

A derived ratio organises reporting without becoming a league table. Reusable-prefix ratio, cached or reused input tokens over total eligible input tokens, computed per provider in its own units, alongside uncached fraction, write-to-read ratio, cost per request and per session, and time-to-first-token reduction. No composite score across providers: their reporting units differ, and a single number would launder the differences this chapter took care to preserve.

The constraint pruning inherits

Return to the session that opened the chapter: 120,000 tokens in context, 80,000 cached cheaply, 35,000 of them duplicates, obsolete output, failed payloads, completed traces. Caching reduced their computational cost. It changed nothing about their membership. The coming deletion chapter must therefore optimise something closer to this than to bare token counts:

benefit of removal
  ≈ reduced occupancy
  + reduced uncached processing
  + reduced interference
  − information-loss risk
  − cache-reuse loss
  − mutation and rewrite cost

The equation is deliberately approximate, a list of variables with signs, not a scoring formula. Its cache terms are this chapter’s bequest: every deletion carries a reuse price, every rewrite a mutation radius, and any pruning result reported without both is unaudited. With that ledger open, the book may finally start removing things:

Which context can we remove without destroying information we still need, and without accidentally losing more cache value than the removal saves?

References

• OpenAI. “Prompt caching.” Official documentation, verified September 2026. GPT-5.6 write/read economics, breakpoints, TTL, routing, telemetry, minimum-length trap. https://developers.openai.com/api/docs/guides/prompt-caching
• Anthropic. “Prompt caching.” Official documentation, verified September 2026. Automatic and explicit breakpoints, write/read pricing, TTL options, minima, lookback, invalidation hierarchy. https://platform.claude.com/docs/en/build-with-claude/prompt-caching
• Anthropic. “Context windows.” Official documentation, verified September 2026. Cached prefixes still occupy the window; cached computation is not freed capacity. https://platform.claude.com/docs/en/build-with-claude/context-windows
• DeepSeek. “Context Caching.” Official documentation, verified September 2026. Disk-backed prefix units, full-match rules, hit/miss telemetry, best-effort lifetime. https://api-docs.deepseek.com/guides/kv_cache
• Moonshot AI. “Use the Context Caching Feature of Kimi API.” Official documentation, verified September 2026. Automatic caching, stability-first layout, append-only tool loading. https://platform.kimi.ai/docs/guide/use-context-caching-feature-of-kimi-api
• Zheng, L., Yin, L., Xie, Z., et al. “SGLang: Efficient Execution of Structured Language Model Programs.” Preprint, arXiv:2312.07104. RadixAttention prefix reuse in open serving systems. https://arxiv.org/abs/2312.07104