# Representation Is Part of Context
# Book: Context From First Principles
# Chapter URL: https://programmer.ie/books/context/18-chapter/
# Prose fingerprint (sha256, normalized): fbbf1f6438df20840a63d96e5f3b6f5632c37c5a72ae87b343d8c05a03ff3b96
# Snapshot generated at build time by Hugo. This is the complete rendered prose
# of the chapter; it is not a transcript of the source file.

Here is an illustrative case built from one fixture. A migration uses PostgreSQL, its identifier is 01947, it remains blocked by test suite A, and the team rejected the SQLite fallback after a staging incident. Render the same declared facts and qualifications once as prose and once as a labelled record. Asked for the unresolved migration’s identifier, a model might locate the field more reliably in the record than the digits embedded in prose. Asked why SQLite was rejected, it might do equally well with either representation if both preserve the reason and its qualification. If the record omits that qualification, the comparison no longer isolates representation: it also tests missing information. That distinction is the problem this chapter must keep visible.
The content object and the experiment’s intended obligations were held fixed, but the concrete inputs need independent validation before we can say that nothing relevant was selected or lost differently. When both representations satisfy the same frozen obligations, differences in model behaviour become evidence about presentation under the tested conditions. When one drops a required qualification, a transformation failure is already present before the model runs. This chapter is about distinguishing those outcomes:
information identity ≠ representation identity
Chapter 12 established that one content object can have several legal representations satisfying different preservation obligations and costs, and that this is a choice and not a fidelity ladder:
representation choice ≠ fidelity level This chapter asks what organisation and encoding within such representations do to computation. Chapter 14 owns what is admitted and Chapter 17 owns what tools expose. Authority, freshness and scope belong to Chapters 19 to 21. The question here is:
Given information that should become model-visible, how does its representation change the operations a model can perform, the distinctions it can preserve, and the resources it consumes?
Information, representation, serialisation flowchart TD U["Source information<br/>or fixture canonical state"] S["Semantic representation<br/>which structure is made explicit"] Z["Serialisation<br/>the concrete model-visible encoding"] T["Model-visible tokens"] M["Model"] U --> S --> Z --> T --> M The first box needs care. For an experiment, the experimenter can define a fixture canonical state, a semantic object from which several renderings are generated. For real information there may be no representation-free object. A source expresses information in its own form, and mapping it into another structure can itself require interpretation:
canonical fixture state ≠ representation-independent truth A semantic representation determines which distinctions, relations and groupings are explicit. A serialisation is the concrete model-visible encoding of that representation. A candidate can carry a serialisation before assembly. The rendered bundle is the later artefact produced when admitted items are ordered, wrapped and combined by the assembler (Chapter 23); its visible separators and labels can themselves affect interpretation. The observation boundary in Chapter 2 may capture the host’s assembled form without exposing the provider’s final rendering, so this chapter does not equate the two. A preservation obligation specifies a distinction that must survive a transformation for a declared evaluation, including obligations that apply beyond the current question.
Semantic families are defined by the property they make explicit, and syntaxes are not families:
Semantic representation Property made explicit Serialisations Narrative Relations carried linguistically Prose, outline Labelled record Field identity Plain key-value text, JSON object, XML record Table Schema repeated across rows Markdown table, CSV, HTML table Relation representation Typed relation and endpoints Line triples, JSON objects, CSV edges Hybrid Several forms, each with a declared operation Any combination Serialisation is not punctuation alone. Moving from JSON to XML can change field order, repeated labels, nesting, escaping, type expression, tokenisation and how salient the boundaries are, so the same semantic schema does not imply the same model-visible cues. That is why it needs its own experiment.
Three refusals keep the layers clean. Representation is not extent: a full record and an exact span of it, or a full document and an exact span of it, differ in extent and not necessarily in representation. A smaller extent is not automatically a lower fidelity. Structure is not correctness: a JSON object can be well formed and wrong, and a paragraph can be unstructured and accurate. Explicit and correct are separate properties:
schema-valid ≠ semantically correct ≠ obligation-complete explicit relation ≠ source-supported relation Inspect this in the Context Viewer The viewer scene for this chapter renders the same migration facts four ways — prose, labelled record, compact note, locator — and checks each against four frozen obligations. Open the scene and read the obligation table before judging any form. What to examine: which obligations survive each rendering, and why the compact note’s dropped qualification is loss rather than compactness. Equivalence here is obligation-relative; no model ran and no behavioural prediction follows.
Equivalence holds only relative to declared obligations. Constructed illustration. Open the representation scene in the Context Viewer
Equivalence needs obligations Saying that two renderings carry the same information conceals a choice: atomic facts, relations, qualifications, provenance, temporal status, uncertainty and operational affordances are different preservation demands. In the opening example, a record that drops the reason’s qualification does not satisfy the same obligations as the prose, even if both retain the migration identifier and headline decision. Such a pair is suitable for testing transformation loss, not for a controlled claim that only representation changed.
Equivalence is always equivalence under a declared preservation obligation.
same atomic facts ≠ same represented information A fact manifest can establish atomic preservation. It cannot by itself establish that every relation or interpretation relevant to a computation survived. The obligations a fixture may declare include exact values, entity identity, relation direction, qualification, status, provenance, temporal ordering, uncertainty, the distinction between null and absent, conflict and source boundaries. No fixture needs all of them.
Two consequences follow. A manifest pass establishes equivalence only with respect to the frozen obligations in that manifest (manifest coverage bounds the equivalence claim), so manifest pass ≠ semantic identity. And a deterministic renderer establishes reproducibility and nothing more, so deterministic ≠ lossless and deterministic ≠ semantically equivalent. Intended lossless is not demonstrated equivalent.
Start with operations, not formats The operations a model may be asked to perform include exact lookup, comparing records, filtering, aggregating, following a relation, explaining a rationale, applying an update, detecting a contradiction and preserving provenance. Before optimising model-facing representation, decide which operations should instead be executed by deterministic runtime code. That choice depends on whether the operation is fully specified, the required source is accessible to code, and the implementation can be checked. Chapter 17 showed that producer-side code can reduce what enters context; it did not establish that every lookup, filter or comparison is better performed outside the model. Optimising representation for an avoidable model step may solve the wrong problem.
For the operations that remain with the model, utility is relational:
Representation utility is operation-relative.
It also depends on the model, task, context length, data size, surrounding bundle, serialisation, required precision and output requirement. No scalar is defined and no representation is declared best without naming the operation.
Some mechanisms are plausible and remain hypotheses. Labelled fields may help exact state, since field identity is explicit. A table may help comparison across repeated records. Typed relations make endpoints and direction explicit for traversal. Narrative can carry a qualified rationale in fewer moving parts. In each case implicit is not absent: prose can state a relation, and triples merely encode relation identity in a fixed structure. The useful mechanism may also be the labelled field and not the braces, which is why key-value text and JSON appear as separate serialisations of the same record.
Structure does not inherently remove nuance. A schema can carry decision, reason, qualification, counterfactual and evidence, and preserve them exactly. What a schema omits it can erase:
schema omission can become semantic omission representation-family capacity ≠ instance preservation A record family that can represent causal qualification is not to blame when one renderer dropped the field.
Structure can also manufacture distinctions. Source prose reading “the service may have failed after migration” converted to failure_cause: migration has turned a tentative temporal association into an asserted causal relation. The generated field is more explicit, but it has changed both the epistemic status and the type of relation. A schema may be perfectly valid while its populated instance is unsupported; the validation must compare claims and qualifiers against the source, not merely parse the result:
making structure explicit ≠ discovering structure Other pieces of vocabulary need boundaries. A graph data model, a graph serialisation and a graph algorithm are separate: a model given triples is not thereby running graph reasoning, and no model receives an abstract graph unless the runtime supports one. Tables spend tokens to encode two-dimensional layout, through headers, delimiters and repeated row structure, and a compact table is not therefore easy to read, since it can add positional dependence, header dependency, null ambiguity, escaping and row-boundary ambiguity: compact ≠ easy to interpret.
What the external evidence establishes None of this was measured on this book’s workload.
Source Evidence type Population/task Verified claim Chapter uses Does not establish Sui et al., “Table Meets LLM” (WSDM 2024) Peer-reviewed benchmark Seven structural-understanding tasks (for example cell lookup, row retrieval, size detection) on GPT-3.5 and GPT-4 Input format, content order, role prompting and partition marks were varied; the authors report gains from structural prompting on tabular datasets Presentation and boundary cues can change table-task performance That markup, HTML or explicit structure is better in general; anything about current models Li et al., “LongTableBench” (Findings of EMNLP 2025) Peer-reviewed benchmark 5,950 instances, seven table formats, eighteen domains, up to 128K tokens, 52 models The authors report that only the strongest models keep robust performance as length and format diversity grow Format effects interact with model, task and length A winning format Lepagnol et al., “Format Matters” (LREC 2026) Peer-reviewed, output side JSON, XML and inline key-value output for slot filling and NER, 13 open-weight models Format choice produced significant swings (2 to 46 F1 points) depending on model and dataset Format, model and task interact even with the semantic target fixed Any input-format advantage Anthropic prompting guidance Vendor guidance Prompt engineering for Claude models XML tags separate instructions, context, examples and inputs; documents can be wrapped with source metadata Explicit boundaries and hierarchy are recommended practice That XML beats JSON, Markdown or prose OpenAI Structured Outputs Vendor API documentation Schema-constrained generation Responses adhere to a supplied JSON Schema; refusals are programmatically detectable A typed application boundary That structured input reasons better MCP draft specification Protocol specification Tool results Results can contain text, image, audio, resource links and embedded resources; structuredContent may conform to an outputSchema, with serialised JSON in a text block for compatibility Practical representation choices exist at the tool boundary That those forms carry equivalent information The conclusion these sources support is narrow. The table-input benchmarks show representation and presentation effects on their tested models and tasks; the output-format study demonstrates a separate output-side sensitivity and cannot serve as direct evidence for input comprehension. Vendor guidance establishes recommended practices, while API and protocol documentation establishes available mechanisms, not behavioural improvements. The studies also vary format alongside other presentation factors in some conditions, so a reported gain is not necessarily attributable to serialisation alone. An MCP resource link points to information outside the returned payload and is not equivalent to returning that information in full.
Fixtures and engineered equivalence Asking a model to rewrite prose as JSON and treating the results as equivalent confounds the experiment with conversion quality, since the rewrite can omit, mutate or add. The fixture inverts the pipeline:
canonical fixture state ↓ task family and preservation obligations (frozen) ↓ deterministic renderers ↓ preservation validation ↓ freeze, then model runs Obligations are frozen before rendering and before any model result is seen. Adding an obligation after inspecting results, because it favours one representation, is a defect.
Fixture validity is assessed before model evaluation and independently of the predicted outcome. Validation checks which frozen preservation obligations each rendering satisfies, whether every compared condition is eligible for the particular contrast, whether the task exercises the intended operation, and whether any rendering accidentally signals the answer. Difficulty and answerability are checked without using the model results whose differences the experiment is meant to explain. The validity decision and exclusion rules are frozen before outcome inspection; a representation that fails a required obligation is recorded as such rather than silently repaired to favour a hypothesis. Then:
unexpected result ≠ invalid fixture If the predicted advantages do not appear, that is a result. Rebuilding the benchmark until the expected split appears would defeat the experiment.
Excluding a condition needs care. In a serialisation comparison, a serialiser that cannot encode the held-constant semantics is ineligible. In a representation comparison, a family that cannot express a required distinction may be exactly what is being tested, and dropping it hides the finding. Cannot encode the control is different from cannot express the task-required distinction.
Cost, exactness and null semantics Measure bytes, characters, model-tokenizer counts when the tokenizer is available, and provider-reported input usage where it can be correctly attributed; label counts as observed, derived, estimated or joined under Chapter 2’s vocabulary. A JSON array of twenty objects with six repeated keys renders those key names up to one hundred and twenty times, whereas a table may render six headers once. That is a structural comparison, not a token saving: field values, escaping, missing-value conventions, row identifiers and parser requirements also contribute. Even a lower measured input-token count is not necessarily a lower trajectory cost when retries, clarification or conversion increase. CSV has delimiters and a header convention, but does not by itself express per-cell types or the domain semantics of an empty field.
representation token cost ≠ trajectory cost if errors, retries, clarification or conversion work rise.
Fixtures for exactness must declare their semantics before a renderer is judged. A fixture might distinguish explicit null (known to be unavailable), an absent field (not supplied), an empty string, and numeric zero. Those meanings belong to the fixture contract, not to the punctuation, and each serialisation needs an unambiguous way to retain them. A leading-zero identifier such as "01947" requires byte-accurate preservation only when the domain treats it as an identifier rather than a number; a numeric conversion that loses the zero is otherwise not necessarily wrong. Type checking therefore needs both syntactic and domain-level criteria:
syntactic type ≠ domain type Provenance and conflict work the same way. A representation can carry explicit provenance fields, and a field present is not provenance verified. It can also preserve disagreement, and normalisation must not silently collapse conflict. Referent ambiguity has two cases that stay separate: a deterministic fixture generates explicit roles from canonical state, where a production conversion from prose requires interpretation and carries risk.
Reading and updating Read accuracy is not update safety. In an update task the model receives a frozen state representation, an observation and an explicit transition rule, then produces the next state. The evaluator checks the intended change, all fields that must remain unchanged, identifiers and relations, legal status transitions, output validity and unsupported additions. Marking a migration verified may require a verification time, a verifier and clearing the blocking reason, but those dependencies must be specified by the fixture; otherwise a model cannot be judged against an unstated rule. A valid local edit is not necessarily a valid state transition, and a valid transition is not necessarily preserved through later edits.
Repeated edits, one, three and five, test whether required state survives, watching for unsupported additions, stale fields, lost relations and provenance, schema validity and cost growth. The target is not an identical string after a round trip. This is representation editing and not compaction, which Chapter 11 owns.
Machine interoperability keeps its own ledger. Parseability is not comprehension, schema validity is not task success, and task success is not integration value. An application may choose JSON for integration even where reasoning ties.
Normalisation is transformation Normalisation is a family of operations over heterogeneous sources, not inherently a semantic rewrite. It may change only the encoding while preserving an existing semantic structure, map source concepts into a common schema, or do both. The semantic-mapping case can change meaning through omitted distinctions, new inferences or mistaken equivalences; a purely syntactic conversion can also fail, but it is not automatically an interpretation. This distinction matters because the experiment must attribute observed loss to the operation that introduced it. The options are not a binary:
Source-native: keep the original representation. Canonical semantic representation: map every source into a shared schema. Canonical serialisation only: keep semantic structure and encode it uniformly. Hybrid: normalise selected fields and retain source-native evidence. Normalisation can create false agreement. Source A says status: blocked and source B says “waiting on tests”, and a mapper writes BLOCKED for both, which may erase a distinction. It can create false conflict too, when done, complete and resolved mean the same thing:
normalised equality ≠ source equivalence source difference ≠ semantic conflict A normalisation may also perform interpretation before the model sees the task. If prose reads “Service A appears to depend on B” and the normaliser writes depends_on(A, B), the improvement is not a pure representation effect, so representation and semantic inference during conversion have to be separated. Normalised data without lineage (source identity, original representation, normaliser identity and recipe, output representation, and unknown mappings) weakens auditability. That is a requirement to state, not a schema to design here.
Hybrids and consistency One plausible production architecture is a hybrid: typed state beside short rationale beside a table of repeated records beside relation triples beside raw evidence by reference. A hybrid is several representation families coexisting because they serve declared operations, and one that cannot say which part serves which operation is a pile. It is a hypothesis and not a winner.
Hybrids create a risk of their own. If typed state, prose rationale, a table and triples all claim overlapping properties, they can diverge through duplicate claims, version skew, inconsistent updates and drift:
representation diversity ≠ free redundancy Where two admitted representations claim to encode one property, disagreement between them has to remain observable and not be silently reconciled. Which one governs is Chapter 19’s question.
Ecological gate Nothing in the empty corpus of real traces supports machinery yet. Before building a representation optimiser or a canonical normaliser, real traces should measure the representation families currently admitted, the share arriving source-native and already structured, the prevalence of repeated-record and relation-heavy candidates, exact-field and null-versus-unknown obligations, conversion frequency and conversion field loss, duplicated forms of one object, hybrid bundles, parse failures, schema mismatches, whether downstream tasks require cross-source uniformity, and the token share representation occupies:
representation can matter ≠ representation choice is a prevalent problem
Proposed experiments Nothing here has been run. Every task family reports separately and none is averaged into one score.
Experiment A: semantic organisation. With a frozen fixture state, preservation obligations and model task, how does making different semantic structure explicit affect operation-specific behaviour and total resource use? Conditions are chosen per fixture family: narrative versus labelled record for state lookup; narrative versus table for repeated-record comparison; narrative versus typed triples for relation traversal; and prose versus structured decision-and-reason fields for explanation, with a hybrid only where its components have defined purposes. Every condition is audited for obligation survival before model evaluation. If a representation cannot express a required distinction, its failure is reported as a representation limitation rather than pooled with a controlled, obligation-equivalent contrast. An oracle is a hindsight comparator that exposes task-relevant distinctions; it is not guaranteed to be a behavioural upper bound, because presentation can still impair use. Task fixtures, renderings and stochastic runs are counted separately. Report accuracy, unsupported claims, exact-field and relation use, editing failures, active tokens and management costs by operation; do not average unlike operations into one headline number.
Experiment B: serialisation. Hold the semantic schema and frozen obligations constant while changing its concrete encoding: labelled records as key-value text, JSON or XML, and tabular data as Markdown tables or CSV. Round-trip parsing verifies schema-level preservation but not behavioural equivalence. Freeze field, row and relation order where possible, together with task instruction, delimiters external to the rendering, examples, answer format and framing. Equivalent encodings can have different token lengths and shift the decisive fact’s location, so report total input tokens, absolute and normalised position, and distance from the question. Do not claim a pure format effect if those cannot be independently matched: identify the result as the effect of the complete rendered encoding. A separate boundary-cue ablation can vary labels or wrappers without conflating it with the primary comparison.
Experiment C: normalisation, only if earned. Run only if the earlier experiments and real-workload observations leave a consequential question about converting heterogeneous sources. Compare source-native inputs, syntactic-only canonicalisation where feasible, deterministic semantic mapping, model-generated semantic mapping, a hybrid retaining recoverable source evidence, and an oracle reference. Audit each conversion before downstream use for omission, mutation, unsupported addition, status, provenance, relation, type and ambiguity handling. Then freeze its output and evaluate the model separately. Report conversion success independently of downstream task success; if conversion changes a source’s meaning, do not credit or blame downstream representation alone. If Experiments A and B leave no meaningful conversion problem, defer C.
What would change the book. If plain labelled key-value text matches more elaborate serialisations across the relevant operations, the format optimiser goes. If model differences dominate format effects, or savings are negligible against other context costs, or runtime operations remove most model-side structured work, the same follows. Canonical normalisation is rejected if conversion errors dominate, source-native wins, canonical forms erase distinctions, uniformity yields no measurable benefit, or hybrids that keep the source recover needed information more safely. Hybrid complexity is rejected if it provides no material operation-specific benefit after accounting for representation cost, duplicated state, consistency obligations and management overhead, reported as separate dimensions and not as one scalar. A null result is useful in each case.
What the compiler renders The Chapter 23 compiler receives candidate representations that have already been produced and validated. It selects and assembles among eligible forms rather than inventing a missing compact representation or remapping a candidate’s semantic structure. Its own wrapper still changes the model-visible presentation: headers, labels, separators, item order and closing text contribute tokens and can affect comprehension. The compiler’s recorded assembly is not necessarily the provider’s final rendering, as Chapter 2 established, and neither the bundle’s declared cost nor its identity guarantees identity of the rendered request. The rendering contract therefore needs its own version and trace:
no semantic transformation ≠ no representational effect deterministic renderer ≠ neutral renderer A fixed separator can affect comprehension, and determinism gives reproducibility and not neutrality. The rendered text is also its own object with its own cost, distinct from the bundle it renders, and Chapter 23 notes that rendered cost differs from the declared sum of item costs. Bundle identity is not rendered-representation identity, which matters for tracing (Chapter 25).
The handoff Representation can preserve source identity, provenance, status, conflict, confidence as metadata and exact source wording where an obligation requires it. It cannot decide which source governs:
preserving disagreement ≠ resolving disagreement Once several well-formed sources share a bundle, form has done what it can. What remains is disagreement, and Chapter 19 asks who judges it.
References Sui, Y., Zhou, M., Zhou, M., Han, S., Zhang, D. “Table Meets LLM: Can Large Language Models Understand Structured Table Data? A Benchmark and Empirical Study.” WSDM 2024. https://arxiv.org/abs/2305.13062 Li, L., Tian, J., Chen, H., et al. “LongTableBench: Benchmarking Long-Context Table Reasoning across Real-World Formats and Domains.” Findings of EMNLP 2025. https://aclanthology.org/2025.findings-emnlp.638/ Lepagnol, P., Ghannay, S., Gerald, T., Servan, C., Rosset, S. “Format Matters: A Critical Evaluation of Output Formats for Prompting LLMs in SLU and NER.” LREC 2026. Output-format evidence only. https://lrec.elra.info/lrec2026-main-593 Anthropic. “Prompting best practices.” Vendor documentation, verified September 2026. https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/claude-prompting-best-practices OpenAI. “Structured model outputs.” Vendor documentation, verified September 2026. https://developers.openai.com/api/docs/guides/structured-outputs Model Context Protocol. “Tools.” Draft specification, verified September 2026. https://modelcontextprotocol.io/specification/draft/server/tools