What a Compilation Proves

The compiler has just returned a bundle of five items for the migration task. It says they were admitted for good reasons. Why should anyone believe it?

The question is about the reach of evidence. A compiler’s output raises a chain of questions: what observations do we have, what propositions do they support, and what remains outside their reach? A passing test is evidence for a particular property on a particular population of inputs, under a test oracle and a set of assumptions. It is not automatically a proof of that property for every possible input. In this chapter, “prove” is reserved for a specified mechanical claim supported by an adequate argument or exhaustive check over a declared finite domain. The broader implementation receives narrower evidence: tested conformance, counterexamples and explicit gaps. None establishes truth, usefulness or universal correctness.

Evidence about a compilation establishes only the properties actually observed, under the assumptions supplied to the check. It does not inherit stronger claims because the mechanism is deterministic or the test passes.

Three questions about a compiler

Chapter 23 separated three properties, and they need separate evidence:

soundness:     when the compiler returns a bundle, does it satisfy the declared constraints?
completeness:  if a legal bundle exists, does the compiler find one?
optimality:    among legal bundles, is the one it chose the best?

No formal proof is claimed for any of them over the compiler’s unrestricted input space. The evidence comprises a finite synthetic corpus, an exhaustive feasibility audit of that corpus, negative tests and a constructed counterexample. The asymmetry matters. For a returned bundle, a checker can test the declared structural constraints against the emitted artifact. For a refusal, checking that no legal bundle exists requires a feasibility argument independent of the procedure that refused. A greedy failure may simply be a search failure:

successful bundle  ≠  proof that no better bundle exists
compile failure  ≠  proof of infeasibility
legal  ≠  minimal  ≠  optimal

Optimality is explicitly not claimed. A reference that costs 670 tokens where a full form costs 400 is already a counterexample to cost optimality.

A trace is an instrument

Start with what the compiler hands back besides the bundle. This is the decision trace for the synthetic case in which a cheap reference needs an expensive resolver, at a budget of 1,400 tokens.

Six candidates, six decisions, each with a recorded reason. (Places are counted from zero, and the table is in bundle order.) The table is a reader-facing projection of the trace, not the complete trace and not an independent reconstruction of the decision sequence. The implementation records a candidate’s class, relevance, marginal cost, budget before and after, dependency closure, admission cause and position as well as its decision and reason. Entries in the underlying trace are sorted by identifier; bundle order is a different order. For a jointly admitted reference and resolver, entries can report the same unit-level budget transition, which should not be read as two separate charges. A failed compilation may leave a partial trace. None of these records is the model’s reasoning, ground truth or a causal account of downstream behaviour; the component is reporting its own actions.

An instrument has to be validated, and the historical version failed that test. Two versions exist. In the frozen Python outputs, the trace copied the budget-before value into budget-after and left every dependency field empty. It could record that the resolver was admitted without recording the dependency that required it. The repaired trace records budget movement and closure, but a populated field is still a producer claim until separately checked. Historical evidence must retain the historical defect rather than silently inherit the repaired trace’s stronger observability. So:

trace exists  ≠  trace is complete
field exists  ≠  field is informative
recorded reason  ≠  correct reason
trace consistency  ≠  world correctness

A trace can show that a stale observation was rejected at the freshness gate, and it cannot show the observation was stale, because the gate was told so. What it makes possible is inspection, which is not correctness.

Levels of checking

Several kinds of check answer different questions, and none is better in general:

The first four concern the compiler. The last three belong to later chapters and are named here so nothing is claimed for them.

What the checker checks

The engine assembles a bundle, and separately implemented validators inspect selected invariants of the result. Their gate predicates are implemented independently: the first version delegated back to the engine’s gate function, so the component effectively marked its own work. Negative tests inject scope, freshness, authority and floor violations, over-budget bundles, missing dependencies and incomplete groups, and check that the corresponding validator rejects them. A second validator inspects result structure and selected trace properties. This is stronger than relying only on the engine’s own assertions, but independence of code is not independence of the specification, of cost assumptions or of upstream verdicts.

Its reach is narrower than “recomputes legality from scratch” suggests:

• It checks the emitted bundle against invariants. It does not compute a legal bundle or decide feasibility, so it cannot say a refusal was unavoidable or a chosen bundle best.
• It shares the data types, the units of cost and, notably, the assumption that two representations of one content identity may not coexist. Chapter 23 found that assumption too strong, and the checker enforces it too.
• Its budget check sums each item’s cost field, which is the declared token count plus an estimate of its decoration, and omits the separator and header estimates the engine also adds, so it is a weaker cost check than the engine’s own. The validator’s budget model, the engine’s render-cost model and a provider’s token accounting are three different things.
• It checks that layout matches item order, and it does not check the order against the policy’s role order.
• It confirms that explicitly named candidates are present, and it does not independently derive which content should have been mandatory.
• Its trace checks cover closure, budget movement and consistency. They do not check that a reason code is right or that a decision is correct.

implementation independence  ≠  specification independence
different language  ≠  independent semantics

Two pieces of code can be separately written and still share one mistaken specification, and they can then agree perfectly while both violating the revised representation model.

Inspect this in the Context Viewer

The viewer scene for this chapter holds the finite corpus and its limits together: 34 bundles and 8 refusals with no audit mismatch inside the corpus, the false refusal outside it, the optimality counterexample, parity as agreement, and the labelled distractors legality still admits. Open the scene and keep the three pieces of evidence apart. What to examine: what conformance establishes, what the counterexample defeats, and what neither says about behaviour. Forty-two matches do not prove a forty-third.

Open the conformance scene in the Context Viewer

The finite corpus

The compiler ships with fourteen synthetic case families, each built to exercise one mechanism and tempt a weaker assembler into the mistake it prevents: budget slack, conflict, dependency cycle, dependency trap, shared dependency, mixed pool, mandatory overflow, no legal representation, required source unavailable, qualification trap, representation alternatives, rendered overflow, stale cheap candidate and wrong scope. Each is compiled at a tight, medium and roomy budget, which gives forty-two compilations. In the current implementation, thirty-four return a bundle and eight an explicit refusal: two of the three budgets for mandatory overflow and all three for each of no legal representation and required source unavailable. Every one matches its expected outcome, and every bundle is within budget in the units the candidates declared.

“Matches its expected outcome” needs a definition of who says the expected outcome is right. Here it is the fixture’s own expected status, a bundle or a refusal, recorded with the fixture. It is expected outcome under the fixture oracle and not independently established correctness, and where several bundles are legal, matching a status is weaker than matching a bundle. The corpus also has case families with no discriminating power for some mechanisms, which the comparison below shows for authority.

Whether the eight refusals are genuinely infeasible is a separate question, and this corpus is small enough to permit an exhaustive audit. The audit enumerates candidate subsets and evaluates them under a stated structural model: supplied eligibility and floors, dependency closure, group integrity, at most one representation per content identity, declared mandatory and required obligations, and the renderer’s estimated cost. All thirty-four returned bundles satisfy that model; none of the eight refused cases has a feasible set in it. This is evidence of no false success and no false refusal within those forty-two tested instances, using the shared cost and exclusivity assumptions. It is neither a proof that the assumptions are appropriate nor a proof of completeness on unseen inputs. In particular, the audit and compiler both inherit the content-identity exclusion that Chapter 23 identified as too restrictive for complementary forms.

Three pieces of evidence are kept apart:

construction experiment:   14 cases × 3 budgets × 6 strategies, historical
finite feasibility audit:  14 cases × 3 budgets, compiler against the exhaustive enumeration
completeness counterexample: one constructed case, outside the corpus

The counterexample is a mandatory item with two forms, a cheap reference that needs an expensive resolver and a dearer standalone form, alongside a required item that also needs the resolver. At the first greedy decision the standalone form has lower marginal cost, so the compiler selects it. At the second decision the required item forces the resolver and the resulting set exceeds the budget. Choosing the reference form initially would let the two obligations share the resolver and fit. The enumerating checker finds that feasible set while the compiler refuses, demonstrating a false refusal under the declared structural model. This establishes that the current greedy procedure is not complete; it does not establish how frequently the defect occurs. The counterexample is described as recorded in the compiler repository but remains exploratory evidence until its fixture, expected outcome and run identity are pinned in the evidence register.

Against simpler assemblers

The construction experiment ran the same forty-two cases through six ways of building a bundle, giving 252 compilations: dump, which fills the budget in arrival order and truncates; top-k by relevance; a weighted packer; hard-gated greedy, which applies the four gates and then ranks; the compiler; and an oracle. The oracle knows from a hidden fixture ledger which items the task needs, and it is a hidden-label selection oracle. It is not a feasibility oracle, and it does not know behavioural utility.

The results are held apart in two tables, because they answer different questions. This is the historical construction result on the earlier implementation, and its counts match the stored summary.

Structural outcomes

Hidden-label outcomes

The claim is narrow: in the historical synthetic construction run, the compiler incurred fewer recorded structural violations than the four specified simpler strategies and returned refusals where the fixture expected them. In the tight mandatory-overflow fixture, top-k returned a 2,061-token bundle after omitting a required constraint and filling part of the remaining space with optional material. Hard-gated greedy avoided the observed eligibility violations yet still broke a dependency and a coupled group; those are distinct failure categories. The counts demonstrate performance against the authored fixture obligations, not a general ranking of assemblers. The weighted packer’s recovery of all must-have items on feasible cases further shows that a single recall measure would not distinguish these structural errors. The hidden-label counts in the second table are neither behavioural outcomes nor proof of harm.

Two limits belong beside the table. The number of authority violations is zero for every strategy, dump included. No case tempts a weaker assembler into breaking an authority rule, so the authority gate is implemented and respected and has never been shown to discriminate:

gate implemented  ≠  gate exercised  ≠  gate discriminates
zero observed violation  ≠  demonstrated mechanism value

And a corpus of constructible cases says nothing about prevalence. That a dependency trap exists does not make dependency traps common, and the synthetic counts are not estimates of real frequency.

Legal is not useful

The compiler admitted five items the ledger marks as distractors and two it marks as harmful. Those labels are authored into the fixture ledger, and they do not show that the items harm a model:

harmful-labelled  ≠  demonstrated harmful
distractor-labelled  ≠  behaviourally irrelevant

In the mixed pool, a fixture item labelled harmful is an instruction-shaped note with relevance 0.55 and a declared cost of 130 tokens. The builder marked it in scope, fresh and authority-eligible, and the compiler admitted it above the relevance threshold because it fit and no enforced rule excluded it. That is an important boundary observation: structural legality is relative to supplied verdicts and constraints. The label does not establish that the content was malicious in an executed model task, nor that the compiler granted it instruction authority. Testing that consequence requires Chapter 19’s control/data distinction and Chapter 26’s behavioural intervention.

Parity, and when parity preserves a bug

The compiler began as part of the research code in Python, was extracted into a package with its behaviour on all forty-two compilations captured as frozen outputs (the result, the trace, the bundle and the rendered text), and was later reimplemented in TypeScript. Three questions were then separate:

behavioural parity: do both emit equivalent outputs on the frozen corpus?
schema parity:      do they serialise the same structure?
semantic correctness: is the shared behaviour the desired behaviour?

The TypeScript implementation matched the forty-two frozen Python cases under the historical conformance comparison: the reported comparison found no semantic or serialisation mismatch and no budget violation in those cases. That is an implementation-parity claim over a finite test set, not a semantic-correctness claim. The six-strategy, 252-compilation construction comparison remains evidence from the earlier Python run and must not be silently presented as a TypeScript benchmark. Parity after repairs has a separate versioned contract because the trace schema changed.

The port had to copy behaviour nobody would have chosen. The token estimator rounds halves to even as Python does, words split on the same whitespace rule, and identifiers compare by character code. It also copied three faults: the budget-after field held the budget before, the dependency slot was empty, and the policy recorded how mandatory content should choose among forms while the engine ignored it. Parity required reproducing all three, and it did:

agreement  ≠  correctness        parity  ≠  validity

Agreement is what a faithful copy of a mistake looks like. Inspection of the trace’s audit purpose exposed fields that were either wrong or empty, while the preserved historical golden files showed exactly what the old reference produced. The repair is reported as leaving selected membership, order and refusal outcomes unchanged; that claim is about selection behaviour, not unchanged trace bytes. The trace and result schemas changed version, and current compatibility compares a documented projection with the older shape while separate tests check the newly meaningful fields. Such a projection can show backward compatibility only for the fields it retains. The mandatory-form policy is now enforced, so any claim of unchanged selection requires verification on the cases to which that policy applies.

A frozen output plays different roles. A historical golden records what the reference did, a compatibility golden fixes a contract, and a normative oracle says what should happen, and a golden containing a defect does not oblige anyone to keep the defect. Differential testing is anomaly detection: a difference means at least one implementation differs, and agreement means agreement on the tested case.

Determinism and identity

Determinism removes one source of run-to-run variation when normalised inputs, implementation, version and policy are fixed. The compiler has no model calls, clock reads, randomness or network access in its decision path, and the corpus checked permutation of candidate arrival order. That supports repeatability for the tested cases and normalisation rules, not universal equivalence for all possible inputs. If two policies are run through the same compiler against frozen requests and candidates, any observed output difference can be attributed to the changed policy within that comparison. Whether the difference is beneficial requires separate behavioural evidence, and an unchanged content hash would still not prove identical delivery.

A bundle’s hash covers each item’s identifier and content, in order:

export function contentHash(bundle: ContextBundle): string {
  const digest = createHash("sha256");
  for (const item of bundle.items) {
    digest.update(item.id, "utf-8");
    digest.update(Buffer.from([0]));
    digest.update(item.content, "utf-8");
    digest.update(Buffer.from([0]));
  }
  return digest.digest("hex");
}

Two bundles containing the same item identifiers and content in a different order produce different hashes under this algorithm. The digest therefore identifies an ordered projection of bundle membership and item text, not the whole compilation. It omits costs, source provenance, policy and compiler identities, trace fields, wrapper text and exact rendered bytes. It also relies on the defined identifier/content encoding and separators; SHA-256 supplies a digest, not a guarantee that two distinct inputs can never collide. Chapter 25’s delivery question requires separately identifying what was rendered and what crossed the transport boundary:

bundle content hash  ≠  full compilation identity
bundle identity  ≠  render identity  ≠  delivery identity
determinism  ≠  cross-version identity

Replay needs the same normalised inputs, policy and compiler version, and the result records only the request and policy versions. The last two identities belong to the next chapter.

What structural evidence shows

The corpus of real sessions is empty, so how often these mechanisms matter is unknown.

Proposed experiments

Nothing here changes the historical construction result. New fixtures would be a new versioned corpus, with the historical 14 × 3 × 6 result preserved as it was.

Validator mutation. Start from independently accepted bundles and alter one property at a time: a scope-, freshness- or authority-ineligible candidate, a below-floor form, a missing named item or dependency, a partial group, budget overflow including separators and header, role-order violation, missing trace entry, wrong closure and wrong budget transition. For every mutation, state which validator owns that rule and whether the existing checker actually implements it. An expected rejection that the checker currently cannot make is a coverage gap, not a passing test. Add unmodified valid controls so an indiscriminately rejecting validator cannot appear successful, and report accepted-invalid and rejected-valid outcomes separately.

Finite feasibility oracle. Pin the enumerator’s code, constraint definition, fixture inputs, expected feasible sets and constructed false-refusal case in the evidence register. Audit the oracle against small hand-checked examples, because exhaustive search over the wrong rules remains wrong. Add complementary representations under a revised, explicitly declared substitute relation; do not force them through the current one-form-per-content-identity assumption and call the result definitive. Add an authority-challenge fixture whose upstream verdict and permitted data role are explicit, plus unknown-cost inputs that the current schema cannot express. Distinguish schema rejection, a structurally invalid emitted bundle and refusal despite a feasible bundle. Report false success and false refusal separately within each declared constraint model.

Trace validity. For each field ask whether it varies when the underlying state varies, whether a known defect can make it misleading, whether it is independently checked and whether it suffices for its audit question, using mutations such as a wrong budget-after, a missing or duplicate candidate and a wrong position.

Parity rerun. Report historical parity and current parity separately, since the trace schema changed.

What would weaken the claims. Structural-soundness evidence weakens if a validator accepts a deliberately illegal bundle, rejects an independently valid control, or disagrees with the finite feasibility audit about an emitted set. One false refusal already defeats completeness for the present greedy procedure, regardless of how many other cases pass. The parity claim must name its schema version and projection, and fails on any unexplained mismatch within that contract. The trace-audit claim weakens if injected false entries remain undetected by the check assigned to them. A broader corpus in which a simpler assembler matches the compiler would narrow the demonstrated advantage; it would not erase the distinct structural properties that were actually exercised.

The handoff

The compiler can be inspected more rigorously than a model, because its transformations are deterministic and its constraints explicit. A trace records its decisions, an independent checker challenges its successful outputs, a finite oracle can test whether refusals correspond to infeasibility, and a second implementation exposes divergence. But agreement can preserve a bug, a checker can share a mistaken specification, forty-two matching outcomes do not prove universal correctness, and a legal bundle need not be useful. The constructed counterexample shows the distinction is not merely theoretical: the current greedy procedure can refuse a request for which its own structural model admits a legal bundle. Unless feasibility is independently established, a refusal proves that this procedure failed to construct a bundle, not that none exists.

Even perfect structural evidence would leave one question unanswered:

Did this exact artifact actually reach the model?

A compiled artifact is not a delivered one, and delivered is not used, and used is not helpful. That is Chapter 25.

References

• Project Context Compiler. Source repository, Apache-2.0, first-party code by this book’s author. The engine, the independent validators, the fourteen synthetic cases with their budgets and the frozen outputs of the earlier Python implementation. https://github.com/ernanhughes/project-context-compiler
• Project Context evidence register (companion repository, evidence/README.md). The pinned construction run and the implementation-parity record consumed here. Structural evidence only, with no behavioural claim.