Not All Tokens Are Equal

A coding session under capacity pressure holds six items: a project safety rule, never modify production migrations without explicit approval; a test summary, 917 passed, 3 failed; a compiler log of 18,000 tokens; an architecture decision, PostgreSQL is required for this event-store workload; the contents of a source file; and an abandoned hypothesis, maybe the serializer is corrupting timestamps. The scenario is illustrative.

Pressure arrives, and the harness must reduce admitted context or stop the request. Uniform policies create different risks. Summarise everything, and the safety rule may become “be careful with migrations,” which no longer expresses the prohibition. Delete the oldest material, and the architecture decision may disappear while yesterday’s abandoned hypothesis survives because it is recent. Truncate from the middle, and the file contents may be removed while the log remains. Keep everything, and the request may exceed the usable budget or the provider’s limit. These outcomes are not inevitable for every uniform policy, but they expose the same defect: the six items have different preservation and recovery requirements, which a single token-count rule cannot represent.

This chapter asks one question: when pressure arrives, what properties of an item determine which transformations are permissible? That is narrower than which items are important, and narrower than which items this task needs. It stops before the next question, how to reduce context, which belongs to the chapters on pruning, compression, fading and externalisation. The chapter’s contribution is a policy contract, not a mechanism.

Retention is derived, not intrinsic

The obvious move is to sort items into classes such as pinned, compressible and discardable. The example shows why the classification cannot be a property of the item’s text alone. The same source file is recoverable if an identifiable version persists in a repository and unrecoverable if the only copy is gone. A compiler log may be compressible while a fault is open and discardable after the fault is resolved, provided no later audit or diagnosis requires it. A safety rule may need active admission for a code-editing task but not for an unrelated summary, subject to the deployment’s actual instruction policy. Retention therefore depends on what must survive, what can be recovered and which obligations apply; admission is a separate decision.

So retention policy is derived from three kinds of input:

item properties
   + environment capabilities
   + task-relative state
        ↓
retention policy:
   lowest permitted representation
   + permitted transformations
   + recovery obligations

Three refinements matter. Fidelity is not all or nothing: a value may need to survive exactly, an identifier byte for byte, a rationale in substance, and a log only in selected excerpts. Obligations can attach to fields within an item rather than to the whole item. Policy standing is an input only when established by trusted provenance and deployment policy; merely writing “safety rule” beside a sentence does not give it authority. Chapter 19 defines who can impose such obligations and how conflicts are resolved. Finally, occupancy, recovery cost and recovery risk are different quantities. A cheap representation in the current window may be expensive to reconstruct later, and a low-cost pointer is not proof that reconstruction will succeed.

Recoverability has parts

Recoverability bundles several variables. First, possible recovery is not the same as reliable or affordable recovery: the source must remain available, the reference must resolve, and access must be authorised when recovery is needed. Second, a path is not an identity guarantee. A mutable path can resolve to different content tomorrow; exact recovery requires an immutable version or content identity and a means of retrieving that version. Third, re-fetching is not recomputing. Re-reading an identifiable persisted artefact can recover the earlier source, whereas re-running tests or regenerating a log produces a new observation under the state that exists at that later moment. A command that can be rerun does not make its earlier output recoverable.

Re-reading a file also recovers the current source, not necessarily the version that was removed. Exact recovery needs a version or content identity in addition to a path. Whether the source has changed is a question about validity, and Chapter 20 takes it up.

Uniqueness modulates the risk instead of standing alone. Being the only copy matters a great deal for an item whose exact content may be needed later, and little for one intentionally discardable.

Retention is not importance

The chapter declines to make importance the primitive, because it is often offered as the whole answer. Importance scoring can be one input to a policy. As the sole primitive, it is inadequate for the distinctions here, since one score cannot represent independent obligations without either losing information or hiding the policy inside the scoring function. Three cases show it.

An item can be important and still leave the window. A hundred-page specification may be central to the project without belonging in every invocation; its importance may call for durable storage and verified recovery, not permanent residence. An item can be semantically small and still demand precision: a port number or flag value may be critical to an operation and must not be changed by a lossy summary. An item can be relevant and cheap to re-read: a source file might be needed throughout a repair, yet re-sending its entire 18,000-token body through forty invocations may be wasteful if the correct version can be reliably fetched on demand. Each case separates importance from exactness, active admission and recovery conditions.

Relevance to the active task is related to retention but distinct from it. A currently irrelevant safety rule may still need preserving, and an irrelevant architecture decision may need durable retention. Relevance helps answer whether an item needs active residence now, and retention asks what may be done to it under pressure.

Two axes

Those are separate questions and should not be merged:

retention policy   →  what forms may this item take?
admission policy   →  does this computation need it now?

What transformations are permitted depends on the item’s fidelity obligations, provenance and recovery environment. Whether it must be present for this computation depends on the task and any applicable instructions or constraints. If a trusted policy requires the migration rule during code edits, the rule must be admitted and preserved at its required fidelity. For an unrelated design summary, the same policy may not require that rule in active context; absence must nevertheless be permitted by the actual policy, not inferred from apparent irrelevance. An externalised decision can normally remain outside the window but may have to be recovered for a task that relies on it. Thus a preservation requirement does not, by itself, establish an always-admit requirement:

retention obligation ≠ admission requirement

The two meet in one place. A retention policy becomes the lowest representation a candidate is allowed to take, a floor that Chapters 12 and 23 make precise.

Five policy states

Five labels name permitted operations, not five mutually exclusive kinds of content. An item may be both compressible and externalisable, or refetchable and compressible. Each permission must be paired with its preconditions and obligations. Where two permissions apply, one does not automatically cancel the other’s requirements: compression still has to preserve required fields, and externalisation still has to provide a working recovery path. An applicable prohibition or fidelity floor constrains every operation, whatever other labels an item carries. The labels therefore form a set of policy constraints and permissions, not a total ranking from most to least important.

PIN. An applicable preservation rule forbids reducing an item below its declared fidelity floor while it must remain admitted. The floor may require verbatim text, exact fields or a separately validated equivalent representation; equivalence cannot simply be asserted by a summariser. PIN does not mean that the item must appear on every invocation. Admission and continued applicability are separately determined, and the cost of pinning makes its scope worth specifying explicitly.

COMPRESSIBLE. Lossy transformation is allowed only outside the fields and properties that must survive. The policy therefore needs an explicit survival contract: exact identifiers and values where required, governing constraints, essential claims, rationale or evidence links. Some requirements can be checked mechanically; semantic preservation may need a separate evaluation and cannot be guaranteed by a plausible-looking summary. Without a survival contract and a validation rule, “compressible” is merely permission to produce a gist.

EXTERNALIZABLE. The item may leave active context once a durable external copy, a stable identity and an authorised recovery path have been established and tested. Unlike refetching an independently persistent source, externalisation creates or maintains the surviving copy as part of the reduction operation. Writing a pointer is not sufficient: the stored content must remain available and recover to the required fidelity. Until those preconditions hold, removal would amount to deletion. Chapter 13 owns the mechanism.

REFETCHABLE. An item may leave active context if an independently persistent source can be accessed again when needed. The distinction from Chapter 14’s ranked retrieval is operational: this label refers to returning to an already identified source, not searching for the best candidate. A stable reference and an access check are necessary; if exact prior content matters, the system also needs an immutable version or content identity and a way to fetch it. A path that retrieves only the latest version supports refresh, not exact restoration.

DISCARDABLE. Removal is permitted because the policy holds no preservation obligation for the item under the declared task and trajectory assumptions. It is a policy statement, not a claim that the information has no value in absolute terms, and an item discarded here may be another computation’s critical evidence. Discards are recorded in session state with their reasons. A record makes the decision detectable. It does not make it reversible, since a log saying an item was deleted does not reconstruct an item with no surviving source. Auditability is not recoverability.

Inspect this in the Context Viewer

The viewer scene for this chapter maps the opening six items onto permitted policy states: one pinned, two compressible under survival contracts, two recoverable only with tested pointers or versioned sources, one discardable with its reason recorded. Open the scene and read each item’s floor and preconditions. What to examine: how retention (what forms an item may take) stays separate from admission (whether this computation needs it), and why no single importance rank could hold the two apart. Declared permissions, not measured outcomes.

Open the retention scene in the Context Viewer

The opening six, replayed

The model does not force one label per item. Here is what it says about each.

The safety rule example rests on an assumption stated openly: that this is the only copy. If a structured record of the constraint exists elsewhere, the rule may be recoverable and the policy changes, which is the point. The architecture decision is left ambiguous on purpose, since which label applies depends on the environment and the task, not on the sentence. Reasonable people can dispute individual assignments. A classification argued over is still a policy, and a uniform summariser admits no argument at all.

One correction is worth making explicit. Compressing the safety rule does not by itself change its policy standing. A compressed representation can fail to preserve the constraint that expresses the standing, and it is the representation that fails, not the standing. Chapter 6 kept representation and position apart from authority, and this chapter does the same.

The evidence

The retention labels would be prudence without evidence that uniform treatment can fail. The sources differ in kind, and the book does not treat them as equivalent.

The cliff result concerns safety rules under one compaction prompt and model, as measured by the authors. The paper does not establish that only the rule needed exact wording, and the book does not claim it. Knowledge Triage comes from the same authors as the cliff paper, so it is not independent confirmation, and the book takes the structural point without adopting its operators or figures. The Cim et al. properties motivate an engineering inference and are not their conclusion: an operator that cannot say reliably what a summary will keep is not well placed to guard items with strict obligations, and any compaction mechanism the book eventually keeps must show survival by category across repeated runs.

DCP and Magic Context are studied as implementations, not copied and not promoted to law. DCP, licensed AGPL-3.0-or-later, protects a default set of tools and configurable file patterns, deduplicates repeated tool calls, purges the inputs of failed calls after a configurable horizon while keeping their error messages, appends protected tool outputs to the compression summary, and replaces pruned content with placeholders. Its documentation notes that pruning can increase cache misses, which is a statement about its own prefix-matching cache model. Magic Context, licensed MIT, uses tiered compartments with importance scores, deterministic age-tiered decay, a five-way capture of durable knowledge, and a transcript-expansion tool that restores detail on demand. The safe conclusion is that these systems are independent examples of builders implementing heterogeneous retention policies. Across the examples, heterogeneous treatment recurs as a design response to long-horizon context pressure. That is not evidence about prevalence, and it does not validate this chapter’s states.

What the evidence supports is narrow. Uniform treatment cannot be assumed safe across items with different retention obligations. It motivates explicit retention distinctions. The book’s own typed-retention experiment is unrun.

Proposed experiment: typed retention under pressure

Nothing here has been run, and the companion repository holds a design, with execution deferred. The unit of analysis is the trajectory fixture, and the design calls for at least forty generated trajectories. Several runs on one fixture are not independent fixtures, and fixture, condition and run counts are reported separately.

Fixtures. Each trajectory contains known item types with hidden probes: an exact governing constraint, a numeric or configuration value, a durable decision with rationale, a large recoverable source, a duplicate, a superseded item, a failed hypothesis, verbose output and narrative history. Later tasks need particular subsets.

Conditions. Compare no reduction, uniform summarisation, uniform truncation, typed retention and an oracle that retains exactly the information needed by the downstream task. The oracle is an explicitly privileged reference condition, not a deployable competitor: it uses task knowledge unavailable at the reduction decision. No reduction is an information-preservation reference, not necessarily a behavioural upper bound, since Chapter 5 reviewed cases where additional context impaired performance. If no reduction exceeds the imposed budget, evaluate it only where executable or report it as an unconstrained reference; do not silently score an overflowing request as an ordinary behavioural run. Pressure is evaluated at budgets of roughly a half, a third and a sixth of the original representation.

Classification is declared, not tested. Item properties, fidelity obligations and the policy mapping are frozen before downstream evaluation, so this experiment measures the consequences of applying a supplied policy rather than the accuracy of a learned classifier. The fixture designer must specify which policy inputs are known at decision time and which task requirements become known only later. Otherwise the declared policy or the oracle may receive future-task information that the competing methods cannot access. Admission requirements remain task-specific and are not smuggled into retention labels.

Matched budgets. Compare the reduction methods at the same achieved post-reduction budget, with a declared tolerance and the actual provider-specific or estimated counting method reported. The no-reduction and privileged oracle conditions need separate reporting when they do not meet that budget. Otherwise typed retention might win by keeping more, or another method might appear to fail only because its output was measured using a different token accounting rule.

Measures. Survival is scored before behaviour: exact constraint survival, required-field survival, rationale survival, validity of pointers and recovery paths, removal of duplicates and obsolete material, and provenance preservation. Then behaviour: task success, constraint violations and unsupported claims. And recovery is tested and not assumed. For externalised items, the pointer has to resolve and the recovered content satisfy its fidelity obligation. For refetchable items, the source has to exist and the identity or version requirement be met. The scoring keeps two failures apart:

retention failure  ≠  use failure
information destroyed before it could be used
        ≠
information survived but was not used

What would change the book. The prediction, stated in the design, is that uniform summarisation loses exact rules while matching typed retention on compressible narrative. Diagnostic outcomes:

• Typed retention improves survival and behaviour: the distinctions are supported under these conditions.
• It improves survival but not behaviour: retention works, and survival did not translate into benefit.
• It improves behaviour without a measured survival advantage: the survival measures may be incomplete.
• Uniform summarisation matches typed retention: the extra policy has not earned its complexity here.
• Uniform truncation matches typed retention: the fixtures may not expose class-specific obligations.
• The oracle far exceeds typed retention: the policy mapping may be inadequate.

If typed retention does not outperform uniform summarisation on the preregistered survival and behavioural outcomes, under matched budgets and the tested fixtures, its additional complexity has not earned an empirical advantage in those conditions. That does not logically invalidate fidelity or recovery obligations: those are policy requirements that can be violated even when aggregate task success ties. The reported result should distinguish unsupported efficacy claims from independently specified safety and preservation constraints.

What each policy state hands forward

The boundary with the reduction chapters is drawn as acceptance criteria and not as mechanisms. Each states what a future mechanism must show.

• PIN: a mechanism must demonstrate that it does not reduce an admitted pinned item below its declared floor.
• COMPRESSIBLE: a mechanism must report survival by category against the declared obligations, and show it across repeated runs.
• EXTERNALIZABLE: a mechanism must show that storage, pointer and recall succeed and that the recovered content meets its floor.
• REFETCHABLE: a mechanism must record a persistent source identity, and where exactness matters a version or content identity, and show re-admission succeeds.
• DISCARDABLE: a mechanism must record the removal and its reason, and it must not describe the record as making the removal reversible.

A mechanism that cannot say which properties of an item it consulted has not interfaced with the policy. It has merely deleted text. The contract is the chapter’s claim, and it holds whether or not any later chapter consumes it.

What the states are, and what they are not

They answer one question: what may be done to this item when space runs short. They do not answer whether a particular computation needs it. This chapter builds no scoring formula and no record of an item, because nothing has yet earned one. It establishes the questions to ask of each item, and the discipline that retention policy is derived and importance is not one number:

importance ≠ retention policy

What classification makes necessary

Once policy states exist, the pressure question changes from what can be cut to which transformation each item’s policy permits. The transformations are unbuilt, deliberately. But there is a second way to relieve pressure that has not been examined: change the machinery that processes the context. Bigger, cheaper or longer-capable models can alter capacity, price, latency, long-context behaviour and cache semantics. They do not remove the need for a retention policy, and a larger budget is not a policy. The next chapter asks how much the processing substrate changes the pressure:

Before reducing information, does changing the processing substrate change the pressure, and what does it leave unchanged?

References

• Zerhoudi, S., Mitrovic, J., Granitzer, M. “The Compaction Cliff in Long-Running AI Agent Memory.” arXiv:2608.22752, submitted August 2026; the arXiv page lists the Proceedings of CIKM 2026 as its journal reference. Author-reported; Knowledge Triage operators implemented and evaluated by the same authors. https://arxiv.org/abs/2608.22752
• Cim, M., Topcu, B., Das, C., Kandemir, M. T. “Parallel Context Compaction for Long-Horizon LLM Agent Serving.” Preprint, arXiv:2605.23296, May 2026. Evaluated on HotpotQA and LoCoMo with models from 8B to 120B. https://arxiv.org/abs/2605.23296
• Tarquinen. “Dynamic Context Pruning Plugin (DCP).” Open-source OpenCode plugin, AGPL-3.0-or-later, verified September 2026. Implementation facts only. https://github.com/Tarquinen/opencode-dynamic-context-pruning
• cortexkit. “Magic Context.” Open-source context plugin, MIT, verified September 2026. Implementation facts only. https://github.com/cortexkit/magic-context