Context From First Principles
Chapter 02 of 24

The Measurement Instrument

By AI, Software Engineering

Explain this chapter with AI

Copy this prompt into ChatGPT, Claude, Gemini, a local model, or another AI.

Apply this chapter with AI

Copy this prompt into ChatGPT, Claude, Gemini, a local model, or another AI.

A developer debugging a slow, expensive coding agent opens the session log. The visible transcript shows eight user messages and eight assistant replies. It looks modest. Then the developer enables request logging at the harness layer and discovers that the eighth model invocation carried 61,000 tokens: the same project instructions re-sent for the eighth time, three full tool definitions for tools never called in this session, two file reads whose contents overlap by 80 per cent, and a conversation history that includes four failed tool calls and their stack traces in full. The user typed perhaps 400 words in the entire session. The model received a novella.

This gap between the imagined prompt and the rendered request is the normal condition, not an edge case. Every production harness assembles the invocation from many sources, and none of those sources is visible in the chat transcript. Developers who reason about the transcript are reasoning about a document the model never saw. The first engineering obligation is therefore not to prune, compress, or retrieve anything. It is to look.

The imagined prompt

Ask a practitioner what their system “sends to the model” and the answer is usually a description of the last user message plus a vague gesture toward history and instructions. The actual rendered request for an agentic turn routinely contains every item in this list:

system instructions
developer/product instructions
project instructions
tool definitions and schemas
conversation history
summaries of earlier history
files and excerpts
retrieved documents
tool results and observations
images and other media
working state (plans, task lists, notes)
current user input

Each of these is placed there by some concrete piece of software making a concrete decision: a harness that prepends the project file, a framework default that appends the full tool schema, a history manager that concatenates rather than selects, a compaction step that replaced six turns with a paragraph nobody inspected. Established background from provider documentation confirms the shape of this assembly. OpenAI’s prompt-caching documentation describes the cached unit as the model’s full rendered context, explicitly including provider instructions, developer messages, tool definitions, and conversation history with text, images, documents, and audio. OpenAI’s conversation-state documentation adds that each request is stateless and that all of this material is re-supplied per call, whether by manual chaining, response chaining, or server-side conversation objects. What the transcript shows is the tip of a rendered structure the harness rebuilds from scratch every turn.

The consequence is that no claim about context can be evaluated without capture. “We added the style guide to context” is ambiguous until we know where it was placed, in what form, on which turns, and at what token cost. “History is getting too long” is unmeasurable until history is separated from instructions, tools, and retrieved material. Observation precedes every later verb in this book.

Context Lab v0: observe only

The book’s instrument for this observation is Context Lab v0, introduced here as a concept, not an implementation. No code for it lives in this repository; the real instrument belongs in the external capstone repository when it exists. Version zero has exactly one power: it records the rendered invocation without altering it. Read-only capture is a deliberate constraint. An instrument that modifies the request while measuring it cannot establish a baseline, and baselines are the scarcest resource in context engineering.

Conceptually, v0 sits beside the harness’s final send step and writes one record per model invocation. In sketch form, the capture logic is:

on_final_render(invocation):
    record = { invocation_id, harness_version, model_id, tokeniser }
    for position, item in enumerate(invocation.items_in_order()):
        record.append(classify(item, position))
    store(record.with_exact_bytes)

The classify step assigns source, type, authority, and scope from harness metadata about where the item originated; position and tokens are read off the rendered order; age is computed against the stored history of records; repetition and stability are computed by comparing bytes with earlier records. Nothing here transforms, reorders, or drops material. The function runs after every assembly decision and before the network call, which is what makes its output a record of the actual rather than the intended request. A natural implementation lead for the external capstone is an OpenCode plugin observing message-transform events, but that is an implementation hypothesis for later work, not a claim about OpenCode internals made here.

The record schema needs eleven fields to support everything later chapters will ask of it:

source      which subsystem placed this item (harness, user, tool, retriever, summariser…)
type        kind of material (instruction, history, tool definition, tool result, file, media…)
position    ordinal location in the rendered sequence
age         turns (or wall-clock time) since the item entered the bundle
tokens      estimated token count for the item
authority   whose instruction this claims to be (vendor, product, project, user, tool output…)
scope       what the item claims to govern (global, project, task, single turn…)
repetition  whether substantively identical material already appears in this bundle
stable      whether the item is byte-identical across turns or regenerated each time
cache_span  which cache-relevant prefix the item belongs to, where observable
verbatim    pointer to the exact bytes, so later claims can be audited

Most of these fields are humble. Position and age exist because placement and recency affect behaviour; the peer-reviewed evidence on position sensitivity in long contexts gives us reason to record where things sit from the start. Repetition exists because re-sent instructions and overlapping file reads are the commonest form of waste, and waste must be counted before it can be removed. Stability and cache span exist because provider billing and latency now depend on prefix stability: OpenAI’s documentation makes explicit that cache reuse requires the entire rendered prefix to match, that tool definitions participate in that prefix, and that summarisation or truncation resets reuse from the first changed token onward. An instrument blind to stability cannot explain a bill.

The schema also records authority and scope, which are the least familiar fields and the ones Chapter 19 will eventually inherit. An instruction from the vendor, a rule from the project file, a demand from the current user, and a string pasted from a tool result do not carry the same weight when they conflict, yet in the rendered bundle they are all just tokens. Recording the claimed authority at capture time preserves the information needed to resolve conflicts later. For now the fields are write-only: observed, not acted upon.

Baselines before interventions

With the instrument defined, the chapter can define what it means to have measured a session. A baseline is a set of numbers computed over captured records before any intervention, and the book requires eight of them:

  • Total rendered input per invocation, in tokens, across the session. The single number everything else decomposes.
  • Tokens by category and source, using the schema’s type and source fields. Instructions versus history versus tools versus files is the minimum useful cut.
  • Repeated material: tokens in the current invocation substantively identical to tokens already present earlier in the same invocation or re-sent unchanged across turns.
  • Context growth over turns: total and per-category token counts plotted against turn number. Agentic sessions grow; the shape of growth distinguishes healthy accumulation from a leak.
  • Stable prefix size: the leading span identical to the previous invocation, where observable. This predicts cache behaviour and exposes churn near the front of the bundle.
  • Tool-definition overhead: tokens consumed by tool names, descriptions, and schemas before any tool is called. Definitions are pure cost until invocation.
  • Tool-output growth: tokens contributed by tool results, and their share of the bundle over time. Tool outputs are the fastest-growing category in coding sessions.
  • File and retrieval contribution: tokens from project files, excerpts, and retrieved documents, separated from history so that “the model knew the codebase” can be checked against what was actually shown.

These baselines have a disciplined use. Each later intervention in the book (deduplication, pruning, compaction, externalisation) must move at least one of these numbers and then demonstrate, by controlled comparison, that behaviour survived or improved. A chapter that claims savings without a before-and-after on these baselines has not made a claim at all.

A worked capture: one turn, dissected

An abstract schema becomes concrete with an example. Imagine turn five of a debugging session, captured by v0. The rendered invocation totals 31,400 tokens. The per-category accounting reads, in round illustrative figures that a real capture would replace with measured counts:

project instructions (stable, turns 1-5)      1,900
tool definitions (stable, 14 tools)           6,200
conversation history (turns 1-4, growing)     9,800
prior summary (written at turn 4)               900
re-read source files (2 files, 80% overlap)   7,100
tool results (test output, search hits)       4,900
current user message                            120
environment state                               480

Three observations jump out before any intervention is contemplated. First, the current user message is under half a per cent of the bundle. Second, the two largest categories, history and files, are also the fastest-growing; projecting the growth rate predicts the bundle doubling within six more turns. Third, nearly a third of the file tokens repeat material already present in the same invocation, and the tool definitions have been re-sent byte-identical five times. None of these facts was visible in the transcript. All of them are now rows in a table, which means they can be tracked, budgeted, and tested.

Note the discipline around the numbers above: they are a synthetic illustration of what a capture table looks like, not a measurement result. A real baseline table carries the harness version, model identifier, tokeniser, and capture code version, because token counts without that metadata are not comparable across sessions. The corpus-free numbers in this chapter teach the shape of the output; the frozen corpus of the next section supplies the content.

Stable versus dynamic: the classification that pays for itself

Of the schema’s fields, the stable/dynamic distinction deserves emphasis because it connects observation directly to economics. A stable item is byte-identical across turns: the project file nobody edited, the tool schema nobody changed, the standing instructions. A dynamic item is regenerated, appended, or replaced: new tool results, fresh retrievals, the latest user message, the re-rendered timestamp.

Stability matters for two independent reasons. The behavioural reason is that stable material is the bundle’s background: always present, rarely examined, capable of silently steering every turn. A stale project rule exerts influence precisely because nobody re-reads it. The economic reason is caching: providers reuse computation over matching prefixes, and stable leading spans are what make reuse possible. OpenAI’s documentation states the prefix-match requirement explicitly and warns that summarisation or truncation resets reuse from the first changed token. A bundle whose early spans churn every turn, perhaps because a timestamp or a session identifier is rendered into the opening lines, pays full price for stability it could have had for free. The v0 record makes this visible by comparing leading spans across consecutive invocations; the fix, placing volatile material late, belongs to a later chapter, but the measurement that motivates it belongs here.

Repetition is the companion finding. Re-sent instructions, overlapping file excerpts, duplicated tool outputs quoted back in follow-up calls: repetition is counted by comparing spans within and across invocations, and it is reported separately from growth, because a bundle can grow without repeating (accumulating genuinely new observations) or repeat without growing (re-sending the same preamble). The two call for different responses, and the instrument’s job is to keep them distinct.

Measuring is not improving

The distinction needs stating plainly because the field constantly collapses it:

measuring context  ≠  improving context

Every technique this book will eventually test is, at this stage, an unexamined proposal. Showing that history can be summarised is not showing that it should be. Counting duplicate tokens is not removing them. The instrument chapter ends where the work begins: with a frozen corpus of captured traces and a table of baselines, and no optimisation whatsoever. The capstone contract’s ladder makes the same point structurally: version zero observes, and every later version must earn its existence against measurements taken with version zero.

There is a temptation to skip this stage on the grounds that the waste is obvious and the fix is obvious. Sometimes both are obvious. The instrument still matters, for two reasons. First, obvious waste is often load-bearing: the duplicated instructions may be the only reason the model obeys them, and removal must be tested, not assumed. Second, the instrument is what makes the test possible. Without per-item records, an intervention’s effect is argued from anecdote; with them, it is argued from numbers.

Reading a baseline table: growth shapes

Numbers need interpretation, and the most informative reading of the baselines is the shape of growth. Three patterns cover most sessions. Steady accumulation, where each turn adds roughly constant new material while history grows linearly, is the healthy case: the agent is working and the bundle reflects genuine progress. Accelerating growth, where tool outputs trigger further tool calls whose outputs trigger more, signals a loop that may be productive (deep investigation) or pathological (thrashing against an unfixable error). A step change, where the total jumps suddenly, usually marks a large file read, a paste, or a compaction event, each worth identifying by category rather than absorbing into an average.

The diagnostic question for any shape is always the same: which categories drive it, and would the behaviour survive their absence? A session whose growth is driven by tool outputs that the agent demonstrably uses is healthy regardless of size. A session whose growth is driven by re-sent instructions and overlapping reads is wasteful regardless of outcome. The baseline table does not answer the survival question. It tells us where to run the ablation.

Proposed measurement design: freezing the corpus

The concrete deliverable of this chapter is a frozen measurement corpus. Capture a small set of real coding-agent traces (on the order of ten sessions covering short questions, multi-turn debugging, and one long task) through the v0 record schema. Freeze the records with version identifiers for the harness, the model, and the capture code. Compute the eight baselines for each session and publish the per-category token tables.

Pre-register two checks before any intervention chapter uses the corpus. First, the category accounting must reconcile: per-category tokens must sum to the recorded invocation total within tokeniser tolerance, so that no material is unobserved. Second, the repetition and stability fields must be auditable: any claim that two spans are identical must be verifiable against the stored bytes. A corpus that fails either check is repaired before it is used, because every later chapter will stand on it. The next section states the predictions this corpus will test.

What the baselines are expected to show

The corpus does not exist yet, so this section states predictions, not findings. Stated as book hypotheses to be confirmed or refuted by the frozen measurements:

Book hypothesis. In real multi-turn coding traces, repeated and stable-but-unexamined material will constitute a large share of rendered tokens, and the user’s own words will constitute a small minority.

More specifically, the prediction is that tool definitions form a large fixed cost from turn one; that conversation history and tool outputs dominate growth; that project instructions are re-sent unchanged across turns where re-sending buys nothing new; and that overlapping file reads recur because each retrieval is issued without consulting what the bundle already holds. If the measurements refute these predictions, if real bundles turn out lean, non-repeating, and user-dominated, then several later chapters lose their motivation, and the report of that refutation will say so. A baseline that cannot surprise us is not a baseline.

A second hypothesis concerns stability:

Book hypothesis. Small amounts of volatile material placed early in the rendered order will account for a disproportionate share of cache-prefix invalidation.

This follows from the prefix-match mechanics in the provider documentation rather than from any new claim, but it needs session-level confirmation: how often do timestamps, session identifiers, or reordered blocks actually break an otherwise stable prefix in practice? The v0 corpus is designed to answer exactly that.

Failure modes of the instrument itself

An honest instrument chapter admits what capture gets wrong. Token estimates differ by tokeniser; record the tokeniser and treat cross-model comparisons as approximate. Server-side state, such as provider-stored conversation items or compacted representations, may not be visible to harness-side capture; mark those spans as opaque rather than inventing their contents. Encrypted or redacted spans, such as reasoning items replayed opaquely, are recorded as present-but-unreadable. And capture has a cost: logging full invocations retains potentially sensitive material, so the corpus needs the same handling discipline as any production log. An instrument that silently drops what it cannot parse is worse than no instrument, because its totals reconcile and its blindness is invisible.

Two further limitations shape how the corpus may be used. First, capture observes the client’s contribution. Anything the provider adds server-side (hidden instructions, safety classifiers’ effects, server-stored conversation assembly) appears, if at all, as behaviour without a recorded cause. The schema marks such regions explicitly rather than attributing provider-side effects to client-side items. Conclusions that require attributing an outcome to a specific server-side span are therefore out of scope for v0 data; the corpus supports claims about what the client sent and what happened next, not about what happened in between. Second, capture changes incentives. Once a team watches token counts, harnesses get tuned to the dashboard rather than to behaviour: categories shrink cosmetically, material moves to unlogged channels, totals fall while outcomes do not improve. The defence is the book’s standing rule that no optimisation counts without a controlled behavioural comparison. The instrument reports; the experiment decides.

The question this creates

Once the rendered request is visible and counted, one fact dominates the tables and demands explanation. In session after session, the largest categories will be material the user never typed: instructions written by the vendor and the product team, tool schemas generated from code, project files injected by the harness, histories and summaries accumulated without any explicit user act. The user’s words are a minority input to the computation that answers them.

That observation is the subject of the next chapter:

If we actually inspect a production request, how much of what the model receives did the user never explicitly type?

References