Examples Are Data, Not Decoration
Examples Are Data, Not Decoration
Chapter 6 gave the program a model boundary. We can now say what the program is, which LM dependency ran it, and whether fallback output contaminated the record.
That still leaves the unresolved problem:
program
↓
no disciplined evidence
↓
"seems better"
The next six chapters build one deliberately small teaching experiment. We will keep the editorial sentence-improvement program, create a dataset, measure a baseline, design a metric, compile candidate programs, and compare them against a held-out case.
The corpus is intentionally too small to support a performance claim. Its job is to make the experimental machinery visible. Later chapters will distinguish a runnable demonstration from evidence strong enough to justify promotion.
The first step is to stop treating examples as anonymous prompt decoration.
1. The tempting shortcut
A handwritten few-shot prompt usually looks like this:
Example 1:
Sentence: She opened the door and then she laughed.
Goal: Sharpen rhythm.
Rewrite: She opened the door and laughed.
Example 2:
...
That may improve a single prompt. It is weak experimental material.
The examples have no stable identity. Inputs and reference answers are mixed together. Provenance is missing. A note such as “this was accepted by the reviewer” can accidentally become part of the text shown to the model.
The deeper problem is that the data does not encode a machine-readable boundary between task inputs, labels, evaluation metadata, and provenance. A human reader may understand those roles; the program and optimization harness should not have to infer them from prose.
A dataset row needs structure:
case_id
program inputs
reference / label fields
evaluation-only metadata
provenance
split
That shape is already visible in Writer’s editorial optimization work. Rows carry example identity, project/book/chapter metadata, domain, original text, candidate text, local context, human decision, reasons, source, and metadata. CoCoder applies the same discipline to repository repair: train and development cases may be optimizer-visible; holdout cases must not be passed into the optimizer.
2. A teaching corpus
The corpus below is constructed for the book. It demonstrates the mechanism. It does not establish production performance.
TEACHING_CASES = [
{
"case_id": "edit-001",
"sentence": "She opened the door and then she laughed.",
"editorial_goal": "Sharpen rhythm without changing the event.",
"local_context": "The scene is tense but not comic.",
"reference_rewrite": "She opened the door and laughed.",
"required_entities": [],
"forbidden_terms": ["screamed", "cried"],
"split": "train",
"source": "book_teaching_corpus",
},
{
"case_id": "edit-002",
"sentence": "Mara set the lantern beside the map before answering.",
"editorial_goal": "Preserve the action and named character.",
"local_context": "Mara is the only named person in the paragraph.",
"reference_rewrite": "Mara set the lantern by the map before she answered.",
"required_entities": ["Mara"],
"forbidden_terms": ["Maria", "Marta"],
"split": "train",
"source": "book_teaching_corpus",
},
{
"case_id": "edit-003",
"sentence": "The result was very very difficult to explain clearly.",
"editorial_goal": "Remove obvious repetition.",
"local_context": "The paragraph explains a failed experiment.",
"reference_rewrite": "The result was difficult to explain clearly.",
"required_entities": [],
"forbidden_terms": [],
"split": "dev",
"source": "book_teaching_corpus",
},
{
"case_id": "edit-004",
"sentence": "Jalen signed the form, and the room became quiet.",
"editorial_goal": "Improve rhythm while preserving the named entity.",
"local_context": "Jalen's signature resolves the scene.",
"reference_rewrite": "Jalen signed the form, and the room fell quiet.",
"required_entities": ["Jalen"],
"forbidden_terms": ["Jason"],
"split": "holdout",
"source": "book_teaching_corpus",
},
]
The future repository-repair mapping has the same data shape, but not every field has the same authority:
| Editorial program | Repository repair |
|---|---|
| sentence | repository issue |
| local context | repository evidence available at decision time |
| reference rewrite | reference patch or behavioral target, when one exists |
| constraints | engineering constraints |
| accepted/rejected edit | human or governance outcome |
A reference artifact can help scoring or diagnosis without becoming the success criterion. In repository repair, a patch can be correct without matching one historical gold diff.
3. Convert rows into DSPy examples
DSPy uses dspy.Example as the row object commonly supplied to evaluation and optimization code. Current documentation shows examples built from keyword fields and marked with .with_inputs(...).
The fields named in .with_inputs(...) are the fields DSPy passes to the program at call time. The remaining fields are returned by example.labels() and can serve as labels or metadata.
That distinction is narrower than a security boundary. .with_inputs(...) controls the program-call interface; it does not promise that every non-input field is hidden from metrics, optimizer code, or other experiment machinery.
import dspy
INPUT_FIELDS = ("sentence", "editorial_goal", "local_context")
def to_dspy_example(row: dict) -> dspy.Example:
return dspy.Example(
case_id=row["case_id"],
sentence=row["sentence"],
editorial_goal=row["editorial_goal"],
local_context=row["local_context"],
reference_rewrite=row["reference_rewrite"],
required_entities=row["required_entities"],
forbidden_terms=row["forbidden_terms"],
source=row["source"],
).with_inputs(*INPUT_FIELDS)
This line is the boundary:
.with_inputs("sentence", "editorial_goal", "local_context")
The program receives only the declared task inputs. A metric can still inspect non-input fields such as reference_rewrite, required_entities, and forbidden_terms.
Be precise about the boundary: source, human decisions, validation outcomes, or other evaluation-only fields may be legitimate experiment metadata, but they should not be passed to the task program when they reveal information unavailable at decision time. For especially sensitive fields, keeping them out of the optimizer-visible Example entirely and joining them later by case_id is safer than relying on .with_inputs(...) alone.
Concrete leakage:
4. Split before optimizing
For the teaching corpus:
def split_examples(rows: list[dict]) -> dict[str, list[dspy.Example]]:
buckets = {"train": [], "dev": [], "holdout": []}
for row in rows:
buckets[row["split"]].append(to_dspy_example(row))
return buckets
splits = split_examples(TEACHING_CASES)
trainset = splits["train"]
devset = splits["dev"]
holdout = splits["holdout"]
Roles:
| Split | Role |
|---|---|
| train | Cases the optimizer may use to construct or improve candidate program state |
| dev | Cases that may be used for candidate selection, tuning, or development decisions |
| holdout | Cases excluded from optimization and development inspection until the frozen comparison |
For this four-row teaching corpus, those names demonstrate roles rather than statistical strength: two train cases, one dev case, and one holdout case cannot establish general performance.
Also, row-level separation is not always enough. Near-duplicates, multiple rows from the same chapter, later revisions of the same source, or temporally related cases can leak the same underlying answer across splits. Real datasets may need grouping, source-revision boundaries, or chronological holdouts in addition to distinct case IDs.
Do not wait until late in the project to introduce leakage discipline. The first dataset should already encode it.
5. Example, demonstration, training case, evaluation case
These terms are related, but not synonyms.
| Term | Meaning in this book |
|---|---|
| example | A structured row, often a dspy.Example |
| demonstration | An example placed into a prompt/program state as behavior guidance |
| training case | An example the optimizer may inspect |
| evaluation case | An example used to measure a frozen program |
The same row can be assigned different roles in different experiments, but its evidentiary status changes with exposure. Once a case has been used as a demonstration, optimizer input, tuning case, or repeatedly inspected development failure, it can no longer serve as independent holdout evidence for the claim that was shaped by that exposure.
What Usually Goes Wrong
| Symptom | Likely cause | How to diagnose it | What to change |
|---|---|---|---|
| Optimizer appears too good | Evaluation information reached the task program or candidate-proposing logic | Inspect example.inputs() and optimizer-visible fields |
Remove post-decision fields from task inputs; isolate sensitive evaluation metadata when needed |
| Cases cannot be traced later | No stable case_id |
Inspect saved rows | Add persistent IDs before running experiments |
| Train and holdout share the same underlying case | Row IDs differ but source/revision families overlap | Compare source hashes, groups, revisions, and near-duplicates | Split by the unit that must remain independent |
| Holdout performance improves after repeated manual tuning | The holdout became a development set | Audit how often failures were inspected and changes followed | Retire the exhausted holdout and freeze a fresh one |
| Examples encode stale workflow noise | Historical data was accepted blindly | Sample rows manually | Add source/provenance and filter criteria |
Conclusion
We gained a persistent example format and an explicit split discipline. Examples are no longer anonymous prose inside a prompt; they are rows whose task inputs, labels, provenance, and experimental roles can be audited.
We removed two assumptions: that any useful example is automatically safe to expose, and that .with_inputs(...) by itself defines the entire leakage boundary.
We now have a tiny corpus with explicit roles. It is enough to build the evaluation machinery, but not enough to say whether the current program is good.
That requires a baseline.
