Don’t Let the Optimizer Cheat

Chapter 15 expanded the program’s access. It can search repositories, retrieve memory, use tools, and maybe explore large contexts programmatically.

That power threatens the experiment.

The central question:

How can an optimization experiment appear valid while the program or optimizer has quietly gained access to the answer?

Cheating is often accidental. A field named validation_result slips into generation inputs. A memory tool returns the historical accepted patch. A retriever indexes a later repository revision. A GEPA feedback string includes the exact gold answer. A developer inspects holdout failures, changes instructions, and evaluates on the same holdout again.

None of those failures requires malicious behavior. They come from giving one phase of the experiment access to information that belongs to another phase.

The experiment therefore needs a firewall around information flow, not merely a train/dev/holdout column.

1. Leakage paths

Common paths:

CoCoder’s current corpus architecture encodes several of these boundaries explicitly. ProgramEvaluationCaseDTO separates generation-time input from outcome evidence, and sanitize_generation_input() removes known outcome fields before historical executions become case inputs. Dataset splitting uses a stable grouped hash so cases with the same normalized input fingerprint remain in the same split rather than leaking duplicate inputs across train and holdout.

Its frozen DSPy protocol adds another layer: the optimizer receives train/development cases only, candidate demonstrations are audited against holdout IDs before evaluation, and the comparison policy, validation contract, dataset membership, model configuration, and prompts remain frozen after holdout evidence is observed.

2. A small firewall

The teaching version is simple:

This is still only a teaching firewall, but it illustrates two stronger rules.

First, construct generation inputs from an allowlist. A blacklist is useful as a second assertion, but it cannot anticipate every future outcome-field name.

Second, isolation must be checked at more than the case-ID level. Different rows can represent the same underlying input, chapter, repository state, or solution family. The candidate_manifest fields above are an application-owned audit contract, not DSPy fields; if demonstration lineage or feedback lineage matters, record it while optimization is running rather than trying to reconstruct it later.

Even this is not a complete semantic firewall. A field called notes can contain the gold answer without using a forbidden key. Deterministic checks catch structural leakage; adversarial tests and provenance audits are still required.

3. Adversarial leakage example

Suppose a dataset conversion accidentally does this:

bad_example = dspy.Example(
    sentence=row["sentence"],
    editorial_goal=row["editorial_goal"],
    local_context=row["local_context"],
    reference_rewrite=row["reference_rewrite"],
).with_inputs(
    "sentence",
    "editorial_goal",
    "local_context",
    "reference_rewrite",
)

The task model now receives the reference answer. An optimizer can appear excellent without learning the intended mapping.

The important detail from Chapter 7 still applies: .with_inputs(...) controls which dspy.Example fields are passed to the task program. It is not a complete optimizer firewall. Train/dev labels can legitimately remain on an example for metrics or optimization, while holdout examples must remain outside optimizer visibility entirely.

The firewall version constructs the task input separately from the label-bearing evaluation row:

The same rule applies to repository repair. A generation-time view may include the issue, target/intervention references, base repository revision, allowed repository evidence, and validation-contract identity. Historical patches, validation outcomes, engineering deltas, comparison recommendations, and human dispositions belong to outcome evidence when those are what the experiment is trying to predict or improve against.

Then attack the firewall deliberately:

4. Holdout exhaustion

A holdout set is not magic. It is an information role.

Strategies:

5. Tool and memory firewalls

Tool permissions should be split by phase and case time:

What Usually Goes Wrong

Conclusion

We gained an experimental firewall around information flow. Task inputs, labels, feedback, demonstrations, retrieval, memory, tools, repository revisions, evaluator state, and holdout inspection all need explicit phase-specific access rules and provenance.

We removed three assumptions: that a split alone prevents leakage, that different case IDs imply independent evidence, and that structural field filtering can catch every semantic leak.

A trustworthy candidate is therefore not one that merely scores well. It is one whose improvement survives a frozen comparison and whose provenance shows that the candidate and optimizer had access only to information allowed by the experiment.

That is enough to consider promotion. It is still not enough to call the candidate a production version.