Sliding-Window Attention

Put a 2,048-token window on Llama-3.2-1B and score it on the second half of a 4,096-token passage. With full attention its score is 3.00 bits per token. With the window it is 7.07, more than twice as bad. Now let the window also keep the very first token of the passage, and the score is 3.35. One token, kept out of 2,048, takes back 91 per cent of the damage.

A window is the most direct way to stop a cache growing: keep only the last \(W\) positions and the cache has a fixed size. Gemma-2 does exactly that on every other layer, and the library’s cache for it, measured here, stops growing at 4,095 tokens on those layers. But the measurement above shows that a window is not something a serving system can switch on for a model. It is a property of what the model was trained to expect, and the part of it that matters most is not the window at all.

This chapter takes a window apart: what it computes, how far information travels through layers that each see only a window, how to hold its cache in a fixed-size ring, what the library does for a model built with windows, what happens to two models that were not, and what it does and does not save in time. One of the results is that the book’s own starter arithmetic overstated the saving by a factor of two.

Where this sits

Chapter 21 reduced the cache by keeping fewer heads. A window reduces it by keeping fewer positions, and the two multiply: Gemma-2 uses both. Chapters 10, 14 and 16 treated the cache as something that only grows or is reclaimed when a sequence ends; a window is a cache that discards inside a live sequence. Chapter 8 measured attention cost against length and Chapter 13 built the tiled algorithm whose point is to skip work; a window is a mask that such a kernel can exploit, and the timing below shows what happens when it does not.

The Inference Lab piece added here is the windowed attention toolkit: window and sink masks, a rolling-buffer cache for a layer with exact equivalence tests, a banded attention that does \(O(TW)\) work, a windowed GPT-2 for measuring what a window does to a model, and the experiments. Chapter 23 takes up long contexts, where all of this is used.

For the Pi series, the point is that the window, the layer pattern and the positions are part of the model artefact, like the head counts of Chapter 21. A checkpoint carries them in its config, and a runtime that ignores them (or invents them) is running a different function.

What a window computes

Query \(i\) may attend to key \(j\) when \(j \le i\) and \(i - j < W\). Over a sequence of \(T\) positions, full causal attention considers \(T(T+1)/2\) pairs and a window considers \(\sum_i \min(W, i+1)\), which is about \(TW\) when \(T \gg W\). The starter chapter illustrated this with 16 positions and a window of 4, reporting 4,096 operations for full attention against 928 for the window. That comparison is wrong in a way worth stating: the 4,096 counted the whole \(16 \times 16\) square (256 pairs), but causal attention never looks forward, so it needs only \(16 \cdot 17/2 = 136\) pairs. Against the 58 pairs of the window the saving is 2.3 times, not 4.4. A window saves nothing on the half of the square that causality already removed. The test suite now pins this down: the mask has exactly that many entries.

At 32,768 positions with a window of 1,024 the ratio is 16.3 (537 million pairs against 33 million), which is the regime where it starts to matter.

How far information travels

“Cannot directly attend” is not “cannot influence”. Layer \(k\)’s hidden state at position \(i\) is built from layer \(k-1\)’s states at positions \(i-W+1\) to \(i\), so after \(L\) layers a token can affect positions up to \(L(W-1)\) later. Mistral 7B’s paper makes this point: with \(W = 4096\) and 32 layers the theoretical span is about 131K tokens.

I measured it on GPT-2 with a window imposed in all 12 layers. I changed the token at position 100 and recorded, for each later position, the largest change in any logit (float32, on the CPU):

With a window of 32 the last non-zero change is at distance 372, as the formula says, and every position beyond it is bit-for-bit unchanged. Inside the reach, the influence is real and falls steeply: three orders of magnitude from distance 30 to distance 250. At a window of 64 the formula’s limit is 756 but the last non-zero change was at 754; the final hop would need every one of the 12 layers to attend to the very edge of its window, and the product of those tiny weights is below float32 resolution. So the reach is a bound, a long way above what a model can use: the influence at the edge is smaller than rounding. A unit test on a three-layer model with a window of 4 checks the same thing exactly: non-zero change at every distance to 9, zero beyond.

A fixed-size cache

If no layer can read further back than \(W\) positions, the keys and values older than that can be discarded. The rolling buffer of the Mistral paper stores position \(p\) in slot \(p \bmod W\) of a cache of exactly \(W\) slots, so that new entries overwrite the ones that have just left the window and the cache stops growing. The implementation is a few lines:

class RingKV:
    def slot(self, position):                       # sinks (if any) are pinned; the rest rotate
        return position if position < self.sinks else self.sinks + (position - self.sinks) % self.window

    def append(self, k_new, v_new, position):
        s = self.slot(position)
        self.k[:, :, s], self.v[:, :, s] = k_new, v_new
        self.filled[s] = True

    def attend(self, q):                            # attention over whatever is in the buffer
        scores = (q_grouped @ self.k.transpose(-1, -2)) / sqrt(d)
        scores = scores.masked_fill(~self.filled, -inf)
        return softmax(scores) @ self.v

The buffer’s slots are not in position order, and that does not matter: softmax is insensitive to the order of its inputs, and each key carries its absolute position in the rotary phase it was stored with. The tests compare a decode step against a masked dense computation for every position of a stream, before and after the buffer wraps, for windows of 1 to 30 and with and without pinned sink slots, in float64; they agree to \(10^{-10}\). The cache’s size, a test confirms, is the same after 8 steps and after 40.

What the library does for a model built this way

Gemma-2-2B alternates local and global attention layers, 13 of each, with a window of 4,096 on the local ones and 4 key/value heads of width 256 (the Gemma 2 paper’s Table 1 and the cached config agree). The library’s cache for it uses a different layer type for the local layers. I ran prefills of increasing length and counted the bytes it held:

The formula \(2 \cdot h_{kv} \cdot d \cdot 2\) bytes per token per layer, with local layers counted at min(tokens, 4,095) and global layers at all, reproduces every figure exactly. (The library keeps one fewer than the window; I did not look into why.) The hybrid cache is 25 per cent smaller at 8,192 tokens, which is the model’s whole trained context, and the saving approaches 50 per cent as contexts grow, since half the layers still grow: by arithmetic, 1.96 GB against 3.49 GB at 32,768 tokens and 7.2 against 14.0 GB at 131,072, contexts this model was not trained for.

Of the other models cached here, only Gemma-2 uses windows. Mistral-7B-Instruct-v0.3’s config has sliding_window null, Qwen3-1.7B’s has windows switched off, and Phi-3-mini-4k lists 2,047 without my having established whether its implementation applies it. Mistral-7B-Instruct-v0.3, from the family whose first model introduced the rolling buffer, has sliding_window null in the cached config (stated above); I can report the config and not the reason.

What a window does to a model not built for one

A serving system cannot choose a window; the model has already chosen. But the question “what if I imposed one?” has a measurable answer, and it is the reason the first paragraph of this chapter is surprising. I imposed windows by attention mask on two full-attention checkpoints and scored them on late positions of passages from the test chapters, with and without a few initial “sink” tokens that stay visible regardless of the window.

GPT-2, 40 passages of 1,024 tokens, positions 512 to 1,023, bits per token (full attention: 4.087):

Llama-3.2-1B, 8 passages of 4,096 tokens, positions 2,048 to 4,095 (full attention: 2.998):

Two separate effects are mixed in the first column and cleanly separated by the second.

The cliff. Without sinks, even a window of half the passage destroys the model: 7.07 bits per token for Llama, against 3.00. A 2,048-token window leaves 2,048 tokens of perfectly good recent context, and it is not enough. What the window removes at those positions is the first token, and the damage is out of all proportion to its content. This is the phenomenon the StreamingLLM paper reports: models allocate strong attention to the initial tokens whatever they contain (the authors call them attention sinks), so the model was trained to use them as somewhere to put attention that has nowhere better to go; remove them and the softmax has to spread that attention over tokens that are not meant to hold it. My measurements show the effect and its size; the mechanism is the authors’ explanation, which I did not test (an attention-weight analysis would).

The real cost of the window. With the sinks restored, what is left is the price of forgetting: with four sinks, 0.22 bits per token for Llama at a window of 2,048, rising to 1.08 at 128. For GPT-2, which has absolute position embeddings and a smaller context to lose, a window of 512 costs 0.10 bits and a window of 64 costs 1.35. These are the numbers a window should be judged by, and they are far smaller than the cliff. One sink token recovered most of the loss (91 per cent for Llama at 2,048) and four recovered 95 per cent.

Three limits on this. First, the windowed models here are full-attention checkpoints with a mask laid over them. Models trained with windows (Gemma-2, Mistral 7B v0.1) learned to cope and I have measured none; the cliff may be absent or different there. Second, the sink tokens keep their original positions: StreamingLLM re-indexes the retained keys so that positions stay inside the model’s trained range, which I did not do, and which matters for a model run past its trained context. Third, these are language-modelling scores. A window forgets, and the experiment that shows what is forgotten, a retrieval task whose answer lies outside it, was not run.

The operational conclusion is about the rolling cache of the previous section: for a model that has not been trained with a window, a pure ring discards exactly the tokens the model relies on most. The RingKV class pins sink slots for this reason; a ring that does not is correct arithmetic and the wrong model.

Where the time goes

A window reduces what attention can read; whether it reduces the time depends on whether the kernel actually skips the unread part.

Decode is the easy case. A decode step reads the cache, and the cache is bounded. Per layer, batch 16, 32 query heads and 8 key/value heads of width 128, in float16, best of three:

Time stays flat once the window is full: at 32,768 tokens the step takes 12.8 times less and the cache is 8 times smaller, which is the 8 times the Mistral paper reports for a 32K sequence (the authors’ figure, and plain arithmetic: 32,768 over 4,096). For a long-context service this is the dominant saving, and it is exact: the ring is the same computation.

Prefill is not automatic. I timed one attention layer over \(T\) tokens (8 heads of width 128, fp16, window 1,024) three ways: PyTorch’s fused causal kernel, the same fused kernel given the window as a mask, and my banded implementation, which processes blocks of \(W\) queries against the \(2W\) keys that can matter:

The window mask makes the fused kernel slower, by 2.3 to 2.6 times: an arbitrary mask takes a general path, and masked-out pairs are still computed and thrown away. My banded form, which really does \(O(TW)\) work, is slower than the fused causal kernel until somewhere between 16,384 and 32,768 tokens, and at 32,768 it is 1.75 times faster, although it does 16 times fewer pairs. The gap is efficiency: the fused kernel does about 2.9 million (query, key) pairs per millisecond per head here and mine about 0.3. A banded form written as separate matrix multiplies and a softmax materialises its scores; a kernel that skips masked tiles, as the FlashAttention family does for windows (the Mistral paper reports changes to FlashAttention and xFormers giving a 2 times speed-up at 16K tokens and \(W = 4096\); the authors’ figure, not mine; Chapter 8 explains why that kernel is not available on this machine), would have both properties. So the accurate statement is that a window offers a prefill saving that grows with length, that realising it needs a kernel built for it, and that imposing a window with a mask on a general kernel costs time instead of saving it.

DeepInfer’s bounded replay

The starter claimed that DeepInfer’s repository “describes sliding-window attention with bounded replay.” Read at the pinned commit, its README lists “SWA bounded replay” as a feature, described as reducing decoder prefill work by replaying only a bounded suffix, with the retained suffix required to cover the sliding window plus the maximum backtracking length. It does not define the window, and it states that a KV cache system is not part of the release. That is a statement about recomputation: when part of a prefill must be redone, a window bounds how much. It describes neither a rolling cache nor a saving I have reproduced, and it is first-party documentation. I cite it for what it says and no further.

What the numbers do not show

• Two checkpoints trained without windows. The cliff and the sink recovery describe GPT-2 and Llama-3.2-1B with a mask, not models trained with windows.
• Language-modelling scores only. No retrieval or reasoning task; the cost of forgetting what lies outside the window on such tasks is untested.
• Positions not re-indexed. StreamingLLM’s treatment, which matters beyond the trained context, was not implemented.
• One short text set. Eight 4,096-token passages and forty 1,024-token ones from this book.
• Random tensors for timings, an unfused banded kernel. The crossover at 16K to 32K belongs to my implementation; a fused tile-skipping kernel would cross earlier, and I could not run one.
• A bounded decode step, not a full serving loop. The ring is exact, but I did not integrate it with Chapter 14’s blocks, where the equivalent is freeing blocks as they leave the window, nor with Chapter 16’s prefix cache, where I would expect a rolling cache to limit reuse to the prefix still in the window (a hypothesis, not tested).
• Gemma-2 at its trained context only. Cache bytes beyond 8,192 tokens are arithmetic.

Inference Lab increment

The Lab now has:

• window_mask, attention_pairs, reach, bounded_cache_tokens, hybrid_cache_bytes: the mask and the arithmetic, the last verified against the library’s own cache to the byte.
• RingKV: a rolling-buffer cache with optional pinned sink slots, equal to masked dense attention at every step.
• banded_window_attention: \(O(TW)\) attention equal to the dense masked version.
• gpt2_logits: Chapter 4’s GPT-2 with a window and sinks, used to measure reach and quality.
• cache, quality, receptive and scaling commands.

The rule this adds: a window bounds what attention reads, not what the model needs. The cache can be bounded only where the model was built for it and only if the tokens it leans on, the first ones, are pinned; the decode saving is real and exact; and the prefill saving is available only to a kernel that never touches the masked tiles.

What you should now be able to do

• Count the pairs a windowed pass computes and say why the saving against causal attention is smaller than against the full square.
• Compute how far a token can influence later positions through \(L\) windowed layers, and say why the bound overstates what a model can use.
• Implement a rolling-buffer cache with pinned sink slots and test it against masked dense attention across the wrap.
• Compute the cache of a hybrid local/global model and check the arithmetic against the library.
• Explain what a plain window does to a model not trained with one, what a sink recovers, and why a position-ordered or sink-less ring is the wrong cache for such a model.
• Say what a window saves in decode and in prefill, and what the kernel must do for the second.

Key terms

• Sliding window: attention restricted to the last \(W\) positions.
• Reach: the farthest distance information can travel through stacked windowed layers, \(L(W-1)\).
• Rolling buffer: a fixed-size cache in which position \(p\) occupies slot \(p \bmod W\).
• Attention sink: an initial token to which a trained model gives strong attention regardless of its content.
• Hybrid attention: local and global layers interleaved.
• Banded attention: computation organised in diagonal blocks so that work grows as \(TW\).
• Bounded replay: recomputing only a bounded suffix of a prefill.

Reader exercises

1. Add sink slots to the Chapter 14 paged cache: free the blocks that slide out of the window but never the first block. How does the block table change?
2. Re-index the retained positions as StreamingLLM does and run Llama-3.2-1B past its trained context. Does the score hold?
3. Replace the language-modelling score with a retrieval task whose answer lies at a controlled distance. At what distance does a window of 1,024 fail?
4. Measure attention weights on the first token in every layer of Llama-3.2-1B. Does the sink explanation hold layer by layer?
5. Write the banded form as a fused kernel, or use a library that has one, and find the new crossover against fused causal attention.
6. Fill a rolling cache from a long prompt in chunks of \(W\), as the Mistral paper describes, and check it against one-shot attention.
7. Compute the hybrid cache of Gemma-2-9B and 27B from the paper’s Table 1 and compare with the all-global cache.
8. Make Chapter 16’s prefix cache aware of a rolling cache: which prefix lengths remain reusable?

References

• Longformer — Beltagy, Peters and Cohan, 2020; local windowed attention with selected global tokens; read on 21 October 2026.
• Mistral 7B — Jiang et al., 2023; sliding-window attention, reach through layers, rolling buffer cache; read on 21 October 2026.
• Efficient Streaming Language Models with Attention Sinks — Xiao et al., ICLR 2024; read on 21 October 2026.
• Gemma 2 — Gemma Team, 2024; alternating local and global layers, window 4096; read on 21 October 2026.
• deepseek-ai/DeepInfer at 774ce244 — pinned first-party repository; README feature note on SWA bounded replay; read on 21 October 2026.

Evidence in this chapter

Implementation status

• Manuscript: content/books/inference/22-chapter.md
• Runnable examples: examples/ch22/sliding_window.py (original mask and operation counts), examples/ch22/window_lab.py (masks, rolling cache, banded attention, windowed GPT-2 and the experiments)
• Tests: tests/test_ch22_sliding.py, tests/test_ch22_window_lab.py (mask equal to the starter’s, sinks, pair counts, reach, the library’s measured Gemma-2 cache bytes reproduced by the formula, rolling buffer against masked dense attention across the wrap with and without sinks, constant cache size, banded attention against dense, windowed GPT-2 equal to Chapter 4 at full window, exact influence cutoff; all on CPU)
• Evidence: research/ch22/sources.md, research/ch22/results.json, research/evidence-register.md
• Verification command: python -m unittest discover -s tests -v

The chapter is a draft with draft = false. It makes no claim about the quality of models trained with windows, retrieval within or beyond a window, or any fused windowed kernel. Chapter 23 turns to long-context inference, where the cache, the kernel and the quality questions of Chapters 21 and 22 meet.