Make the Window Bigger

Seven chapters have treated the model as fixed ground and asked how the surrounding system should manage information against it. The natural objection has been waiting since Chapter 4: if context is scarce and difficult to manage, why not simply make the window much larger? A team that moves its agent onto a million-token model watches several selection problems relax at once. Histories that needed pruning now fit. Files that needed triage can all be included. The compaction schedule gets quieter. For a while, context engineering looks like a stopgap awaiting cheaper capacity.

Take that response seriously, because it is partly right. A larger, cheaper window genuinely changes the trade-off, and dismissing it would be ideology, not engineering. But the objection as stated confuses two different achievements: exposing a large number in an API, and making a million tokens computationally practical to process and behaviourally usable once processed. This chapter investigates what the second requires. It changes abstraction level deliberately, from the runtime that assembles context down to the model-side computation that consumes it, and it returns with a boundary the rest of the book depends on.

A fourth question, carefully introduced

The book has worked with three questions. Capacity asks how much the computation can contain. Content asks which information enters. Representation asks how that information is expressed. Chapter 8 adds a fourth, but it is not a fourth member of the same family. It belongs to the other side of the interface:

COMPUTATION
What work must the model perform
to make use of the context it receives?

Capacity, content, and representation describe properties of the context problem. Computation describes the model-side mechanism operating over whatever context arrives. The distinction matters because the two sides answer to different owners and different economics. Picture it as the book’s first full-system diagram:

              CONTEXT SYSTEM

     runtime side              model side

 what should enter?       how is it processed?
        │                         │
 selection                    attention
 ordering                     recurrence
 retention                    compression
 externalisation             KV representation
 retrieval                   positional encoding
        │                         │
        └──────────┬──────────────┘
                   ↓
                behaviour

Everything left of the divide is Chapters 1 through 7 and 9 through 24. Everything right of it is this chapter. Behaviour at the bottom answers to both, which is why neither side can declare victory over the other. A cheaper right side relaxes the left side’s constraints without removing its decisions, and the chapter’s job is to make that sentence precise.

Why long context is computationally expensive

Start from the cost structure that makes a bigger window hard. In ordinary attention, each token’s representation is computed against the tokens before it, so the interactions grow much faster than the sequence itself. Doubling the context roughly quadruples the interaction work, the qualitative content of the familiar quadratic scaling claim. The notation deserves its standard cautions: grouped-query attention, FlashAttention-style tiling, mixture-of-experts sparsity, the prefill/decode split, and hardware all change realised costs, so asymptotic curves must never be read as billing or latency schedules. What the notation reveals, and what survives every implementation refinement, is the direction: longer sequences cost disproportionately more work per token, and the disproportion compounds exactly where agents live, at hundreds of thousands of tokens.

Memory tells the same story in a second currency. To generate each new token without recomputing the past, inference keeps the key and value states of everything seen so far, the KV cache. Textbook chain of the chapter:

more tokens
    ↓
more key/value state
    ↓
more memory movement and storage
    ↓
higher long-context cost

At modest lengths this state is an afterthought. At a million tokens it is the dominant resident of the accelerator, and the engineering of long context is largely the engineering of this state: how much of it to keep, at what fidelity, and which parts deserve expensive interaction.

The two currencies clarify a useful split this book will reuse. Processing the existing context, the prefill phase, is dominated by interaction work and sets time-to-first-token. Generating subsequent tokens, the decode phase, is dominated by reading the accumulated state back, and sets time per output token. Long contexts punish both, through different mechanisms, which is why architectures attack both. The deeper serving mechanics belong to the future Inference book; here the split earns its single paragraph by explaining why no one fix suffices.

Three ways of making context larger

“Long-context architecture” hides at least three distinct achievements, and the chapter separates them before examining any implementation:

1. Extend positional range:
   let the model operate coherently at positions
   far beyond its training regime.

2. Reduce state cost:
   store the history of a long sequence
   in less memory.

3. Reduce interaction cost:
   spend expensive pairwise computation
   on a fraction of what full attention would touch.

A large advertised window can result from any combination, and the combinations behave differently. Positional scaling changes how the model handles positions. State compression changes how history is stored. Sparse interaction changes which positions receive expensive computation. Recurrent mechanisms change how history is accumulated and queried in the first place. The decomposition outlives any vendor’s implementation, so the chapter organises by mechanism and invites providers inside only as worked examples.

Positions are not capacity

Every current open model inherits its sense of order from rotary position embeddings, the RoPE construction of Su et al., which encodes absolute position through rotation matrices while expressing relative dependency inside attention, with useful flexibility across sequence lengths. Flexibility is not immunity. A model trained at one length and asked to operate far beyond it meets positions its parameters never learned to handle, and behaviour degrades for reasons that have nothing to do with attention cost. Extending the range is therefore its own problem, distinct from making attention cheaper.

YaRN, the method of Peng et al., is the canonical answer in open tooling: interpolate the rotary dimensions so that positions beyond training map back into the regime the model understands, with authors’ claims of an order of magnitude fewer extension tokens and demonstrated extrapolation past fine-tuning lengths. Qwen’s documentation makes the resulting distinctions concrete enough to build policy on, because Qwen separates what many vendors blur. The April 2025 Qwen3 generation trains natively at 32,768 tokens; its model card for Qwen3-8B states 32,768 natively and 131,072 with YaRN, configured explicitly through a scaling factor of four against the original maximum, with a documented warning that static scaling applies the same factor at all lengths and can degrade short-text behaviour. The 2507 generation moves the native regime itself to 262,144 tokens in split instruct and thinking variants, extendable toward a million per its model cards, and its serving documentation shows the configured maximum as a deployment flag set against available GPU memory, not a property of the weights.

Four phrases, four different objects:

trained context length
        ≠
configured maximum
        ≠
extrapolated context
        ≠
effective usable context

Trained length is what the parameters learned. Configured maximum is what the serving flag permits. Extrapolated context is what positional scaling reaches. Effective usable context is what Chapter 5’s sense of the term demands: demonstrated retrieval, reasoning, and distractor resistance at length. Configuring a larger maximum creates none of the other three, and Qwen’s explicit separation, native figures beside YaRN figures beside serving flags, is the honest template every vendor page should follow. Positions are the cheapest part of the long-context problem to fix and the easiest to mistake for the whole of it.

Compress the state

If positions answer where tokens sit, state compression answers what it costs to remember them. The KV cache grows with the sequence, so architectures that shrink each token’s stored footprint directly shrink the million-token memory bill. DeepSeek’s lineage is the worked example, and it arrives in two stages worth distinguishing.

The background stage is multi-head latent attention, MLA, the V3 generation’s mechanism for compressing keys and values into latent vectors shared across query heads. The foreground is V4’s hybrid attention, documented in the team’s April 2026 technical report with open weights and an open inference implementation. Two mechanisms interleave across the layer stack. Compressed Sparse Attention first collapses each small block of tokens, four in the reference configuration, into one compressed KV entry, then applies sparse top-k selection over those entries through a learned indexer, while a sliding-window branch preserves the most recent tokens uncompressed. Heavily Compressed Attention compresses far more aggressively, on the order of 128 to 1, and then attends densely over the shortened stream without further selection. The report’s headline efficiency figures, for V4-Pro 27 per cent of the single-token inference FLOPs and 10 per cent of the KV cache of its V3.2 predecessor at a million tokens, are vendor-reported on vendor hardware and cited here only to show the mechanism moves the needle it claims to move, not as independently established benchmarks.

The first-principles trade is now statable without mathematics:

retain every token at full fidelity
                ↓
expensive

compress distant state
                ↓
cheaper

but

compression changes what information
is directly available to attention

Readers of Chapter 7 will hear the echo deliberately. A compressor that pools four tokens into one entry has made a retention decision: fine detail survives only through the pooling function’s mercy. This chapter does not claim the model implements PIN or COMPRESSIBLE classes. It claims something narrower and load-bearing for later: model-side compression is lossy transformation of history under a fidelity policy the runtime never sees and cannot audit per item. Whatever the runtime’s retention classes demand, the model’s compressor may independently degrade, and no runtime keep-list reaches inside it.

Attend to less

Sparse attention attacks the interaction term rather than the storage term. Full attention lets each query potentially interact with very many prior positions; sparse attention selects a subset for expensive computation and spends little or nothing on the rest. DeepSeek’s sparse lineage runs from the V3.2 generation’s DeepSeek Sparse Attention, added through continued training atop the MLA base with top-k selection over latent entries, into V4’s compressed variant, where selection operates over already-compressed blocks so the indexer’s search space shrinks with the sequence it searches.

The conceptual payoff is a second selection problem inside the model, and the chapter insists on keeping it distinct from the first. The runtime asks which information should enter the model. Sparse attention asks which internal positions should receive expensive computation. The two rhyme and must not be merged. A runtime that admits a critical safety instruction and a sparse mechanism that never selects its compressed block produce the same observable failure, an unused instruction, through entirely different causes, diagnosable only by different instruments. Chapter 5’s recovery-versus-reasoning split gains a third fork here: recovered by the bundle, selected by the indexer, used by the reasoning. The book’s experiments cannot yet separate the second fork without model internals, which is honestly recorded as a limitation rather than solved by definition.

Selection inside the model also inherits the pathologies of selection anywhere. An indexer learns which blocks look worth attending, and look-worth-attending is a heuristic with failure modes: literal overlap attracting selection the way lexical matches attracted retrieval in Chapter 5, distant-but-decisive blocks starved because nothing in their compressed signature advertises them. The report’s sliding-window branch is best read as structural acknowledgement of this risk: recent tokens bypass selection entirely, guaranteeing that at least the immediate past is never deselected. Locality as backstop, not locality as principle, and the next section explains why the distinction matters.

Remember through recurrent state

Kimi’s answer changes the shape of history itself. In standard attention the past accumulates: every token appends state, and the state grows with the sequence. Kimi Delta Attention, the core of the Kimi Linear architecture documented in the team’s 2025 technical report and open repository, instead maintains a bounded recurrent state updated token by token under a learned delta rule with fine-grained gating, refining the earlier Gated DeltaNet construction. Conceptually:

ordinary growing history

token 1 → token 2 → token 3 → … → token n
                    ↓
          growing attention state

versus:

token
  ↓
update bounded state
  ↓
fixed-size recurrent representation
  ↓
next token

The economic consequence is immediate: state cost stops growing with length, which is why the team’s report claims up to 75 per cent KV-memory reduction and up to sixfold decoding throughput at a million tokens, figures cited here as vendor-reported mechanism evidence rather than independently established benchmarks. The expressivity question is why Kimi does not stop there. A bounded state must forget, and what it forgets is governed by learned gating rather than by any runtime retention policy. Hence the hybrid: Kimi Linear interleaves KDA layers with global MLA layers at roughly three to one, and the flagship K3 generation composes 69 KDA layers with 24 gated MLA layers in a 93-layer, 2.8-trillion-parameter model holding a million-token window. The hybrid’s logic mirrors the chapter’s own structure:

recurrent mechanism
        ↓
cheap persistent long-range state

global attention
        ↓
richer direct token interactions

hybrid
        ↓
attempt to retain both

Attempt is doing honest work in that diagram. The hybrid does not abolish the trade between memory and fidelity; it prices the two differently per layer. Nothing in the public material establishes that bounded-state layers preserve arbitrary distant detail, and the chapter claims no such thing. What the hybrid establishes, and all the chapter needs, is that long context can be exposed through fundamentally different computation than pairwise attention over a growing past.

The locality pattern

Across otherwise unrelated designs, one pattern recurs: recent tokens get high-fidelity treatment while distant tokens get the cheap path. V4’s sliding-window branch keeps the immediate past uncompressed beside both compressors. Recurrent states naturally emphasise the recent through update dynamics. Serving practice preserves fresh tool results at full resolution while old history is summarised first. The information-theoretic excuse is plausible: recent tokens carry current syntax, local dependencies, the immediate tool result, the open turn, while very old information often tolerates coarser computation.

Plausible is not equivalent, and the chapter marks the gap. Model-side locality is positional: it favours the recent regardless of content. Application-level retention is semantic: it favours the governing regardless of age. A critical safety instruction from turn one is old and therefore cheap-pathed by every locality mechanism, while a trivial observation from the last turn rides the high-fidelity branch. The architecture does not know that the old token governs and the new token chatters; it knows only their distances. This is the precise sense in which the model embodies different treatment of near and distant information without implementing retention policy. Chapter 7’s classes classify by what information is. Locality tiers classify by where it sits. The two orderings correlate in practice and coincide never by guarantee.

Train at the lengths you intend to use

Architecture sets what is computable. Training sets what is learned. The V4 report documents pretraining past 32 trillion tokens with a post-training pipeline aimed at long-horizon and agentic behaviour; Kimi Linear documents a 5.7-trillion-token regime behind its checkpoints. The numbers matter less than the principle they instance:

architecture supports a length
        ≠
model was trained extensively at that length
        ≠
model reliably reasons across that length

The third line is Chapter 5’s conclusion reused without re-arguing it. A million-token configuration establishes capacity. Reliable retrieval, reasoning, uniform positional use, and distractor resistance at that length are behavioural claims requiring the kind of evaluation Chapters 5 and 23 describe, and the report’s own vendor-reported retrieval curve, holding strong through hundreds of thousands of tokens and softening toward the million-token mark, reads as consistency with that demand rather than exemption from it. Extension methods sharpen the point: YaRN reaches further by scaling positions, but reaching is not learning, and Qwen’s own warning that static scaling can degrade short-text behaviour shows the bill arriving in a different currency. There is no architectural move that purchases trained competence without training.

The closed-model boundary

OpenAI and Anthropic expose very large windows through production models, and their runtime behaviour, limits, caching semantics, and compaction features are legitimate evidence throughout this book. Their attention architectures are not. No first-party technical documentation from either vendor, as verified for this chapter, describes sparse attention, linear or recurrent state, compressed KV schemes, or positional extension machinery inside their models, so this chapter asserts nothing about them. Their public interfaces expose capacity and runtime behaviour with enough detail for budget and management reasoning, which is what the runtime-side chapters use. The model-side comparisons here rest entirely on architectures whose reports and implementations are public: DeepSeek, Kimi, Qwen. That asymmetry is epistemic discipline. A chapter that inferred closed internals from capability would trade its authority for completeness, and completeness about the wrong object is not completeness.

What architecture cannot decide

The chapter can now state its negative result, the sentence the rest of the book stands on. Suppose architecture made a million tokens nearly free to process, with perfect retrieval at every position and no degradation at any length. The runtime would still face questions no attention pattern answers:

Which instruction is authoritative?

Which information is stale?

Which duplicate is redundant?

Which project does this fact belong to?

Which contradictory source should win?

Which giant tool output is worth retaining?

Which information should sit near the current task?

Which external artifact needs to return now?

Authority is not positional. Staleness is not distance. Redundancy is not similarity of compressed signatures. Project membership is not adjacency. Contradiction is not resolved by attending harder to both sides. Retention worth is the subject of Chapter 7, not of any indexer. Placement near the task is Chapter 6’s canonical layout, not a sliding window. Recall timing is Chapters 13 and 14, not recency bias. Each question belongs to the runtime side of the opening diagram, and each survives every model-side improvement this chapter surveyed, because each concerns what the computation should receive rather than how cheaply it can be processed. Hence the boundary in its strongest form:

long-context architecture
        ≠
context management

The chapter refuses the opposite error with equal force. None of this makes long context useless or architecture irrelevant. Cheaper processing moves real constraints: histories that required aggressive triage fit comfortably, selection thresholds relax, compaction schedules quieten, latency budgets stretch. The relationship is complementarity, stated once and kept:

better architecture
        ↓
changes the cost curve

better context management
        ↓
changes what information is presented

together
        ↓
better context systems

First-party practice already shows the two levers pulled together. Kimi’s own K3 evaluation notes adopt a context-compaction strategy triggered at 300K tokens and report improved task scores with management on than with the full million-token window unmanaged. A frontier architecture with a million-token window, built by the team that best understands its internals, still manages runtime context. Model-side efficiency did not eliminate runtime-side selection in the hands of its own builders. That single documented fact does more work than any argument in this chapter: the distinction is not philosophical. It is operational, and operators observe it.

Two caches, not one

One confusion must be killed before the next chapter, because Chapter 9 dies of it otherwise. This chapter has discussed the KV cache throughout: the model’s internal inference state, keys and values retained so generation need not recompute the past. Chapter 9 will discuss something related but different: provider API prompt-prefix caching, where a serving system reuses computation across requests whose rendered prefixes match. The two share mathematics and vocabulary and differ in ownership, granularity, and control. The KV cache lives inside a forward pass and is rebuilt whenever the runtime sends a new request. The prompt-prefix cache lives in the serving layer and survives across requests exactly when the runtime holds the prefix stable. Model compression shrinks the first. Canonical layout protects the second. A team that hears “caching” and stops distinguishing will optimise one while breaking the other, most often by pruning tokens to save KV memory and invalidating the prefix cache in the same edit. The chapters that follow keep the two ledgers separate. So must the reader.

Proposed experiment: capacity against effective behaviour

The book needs its own measurement even though it will never train a foundation model. The experiment tests the chapter’s central distinction, configured capacity against effective behaviour, using open weights where every regime is inspectable.

Qwen is the primary candidate precisely because its public tooling separates the regimes this chapter names. Take a single Qwen3 dense checkpoint and serve it in three configurations: native regime at or below its trained length, YaRN-extended regime at roughly four times native, and near-limit extended regime approaching the documented maximum. The task is a frozen evidence bundle of the Chapter 5 family: fixed critical evidence, fixed distractors, externally checkable outcome, run at increasing total lengths within each configuration. Measure task success, critical-evidence recovery, reasoning-after-recovery as a separate score, latency with time-to-first-token reported apart from per-token time, memory footprint where the serving environment exposes it, and rendered input tokens. Pre-register the reading: the native regime establishes the behavioural baseline; the extended regimes test whether configured reach preserves it. Any gap between configured maximum and effective behaviour at matched lengths is the chapter’s distinction made numeric, and the YaRN static-scaling caveat predicts where short-text controls should also be run: extension is not free even below the limit.

A second branch compares computational mechanisms rather than configurations, contingent on practicality. A small Kimi Linear checkpoint beside a comparable full-attention open model, run over the same frozen bundles at increasing lengths, measuring runtime and state behaviour, memory footprint, time-to-first-token, and the same behavioural scores. The goal is mechanism demonstration, not ranking: two architectures exposing long context through different computation, with cost curves and failure signatures compared rather than averaged into a winner. Include this branch only if checkpoints fit realistic hardware, the serving stack exposes the required measurements cleanly, and the comparison holds task, lengths, and decoding fixed. Otherwise record it as a deferred optional demonstration. The book’s claim does not depend on it; the primary branch already tests what the chapter asserts.

Both branches share the chapter’s honesty constraints. No vendor benchmark is reproduced or contested. No closed model appears. Efficiency figures reported by vendors are treated as hypotheses about the cost curve, not as measurements of it. What the experiment establishes, if run, is modest and exactly sufficient: where configured capacity ends and effective behaviour begins, for inspectable models, under controlled bundles.

The waste that cheapness leaves behind

Architecture has had its hearing. Suppose it succeeds completely: million-token processing at trivial cost, faithful retrieval everywhere, reasoning intact at every depth. The runtime assembles Tuesday’s bundle: instructions, project rules, tool definitions, accumulated history, retrieved files, fresh observations, the current request. Wednesday’s bundle contains nearly all of it again, plus one new turn. The model recomputes the shared prefix from scratch, burning the same work twice because nothing in the interface remembers Monday’s computation. Capacity solved what to include. Computation solved what it costs to process. Neither solved the repetition, and the repetition grows with every turn of every long session. That is the next chapter’s subject:

Even if the model can process the tokens efficiently, repeatedly processing identical context can still waste computation.

References

• DeepSeek-AI. “DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence.” Vendor technical report, arXiv:2606.19348, April 2026. Hybrid CSA/HCA; 27% FLOPs and 10% KV cache vs V3.2 at 1M (vendor-reported); 32T+ training tokens; open weights and implementation. https://arxiv.org/abs/2606.19348
• DeepSeek. “DeepSeek-V4 Preview Release.” API announcement, April 2026. V4-Pro/Flash sizes; 1M default context; token-wise compression plus DSA. https://api-docs.deepseek.com/news/news260424/
• Kimi Team. “Kimi Linear: An Expressive, Efficient Attention Architecture.” Vendor tech report, arXiv:2510.26692, November 2025. KDA over finite-state memory; hybrid KDA/MLA; open kernel and checkpoints. https://arxiv.org/abs/2510.26692
• MoonshotAI. “Kimi-Linear.” Official repository, verified September 2026. 3:1 KDA-to-MLA hybrid; 48B/3B at 1M; 5.7T training tokens. https://github.com/MoonshotAI/Kimi-Linear
• MoonshotAI. “Kimi-K3.” Official repository, verified September 2026. 69 KDA + 24 gated MLA layers; 1M context; compaction-at-300K evaluation practice. https://github.com/MoonshotAI/Kimi-K3
• Qwen Team. “Qwen3-8B model card.” Hugging Face, verified September 2026. 32,768 native, 131,072 with YaRN; static-scaling caveat; output reservation in defaults. https://huggingface.co/Qwen/Qwen3-8B
• Qwen Team. “Qwen3 documentation (Quickstart).” Verified September 2026. 2507 variants at 262,144 serving flags; thinking-budget mechanism. https://qwen.readthedocs.io/en/latest/getting_started/quickstart.html
• Peng, B., Quesnelle, J., Fan, H., Shippole, E. “YaRN: Efficient Context Window Extension of Large Language Models.” Preprint, arXiv:2309.00071. RoPE interpolation and extrapolation method. https://arxiv.org/abs/2309.00071
• Su, J., Lu, Y., Pan, S., Murtadha, A., Wen, B., Liu, Y. “RoFormer: Enhanced Transformer with Rotary Position Embedding.” Preprint, arXiv:2104.09864. Rotary embeddings; relative dependency; length flexibility. https://arxiv.org/abs/2104.09864
• Burtenshaw, B. “DeepSeek-V4: a million-token context that agents can actually use.” Hugging Face blog, April 2026. Third-party mechanism gloss; figures from the vendor report. https://huggingface.co/blog/deepseekv4