Few-Shot Optimization
Few-Shot Optimization
Chapter 10 established the compilation boundary. The optimizer may propose a candidate program. It may not promote itself.
Now we inspect the first mechanism: few-shot optimization.
The question is practical:
Can we improve the program by selecting or constructing better demonstrations rather than rewriting the program ourselves?
A demonstration is attached to a predictor, so its fields need to make sense for that predictor’s signature.
For a one-step rewrite predictor, a demonstration might look like:
1. What BootstrapFewShot does
Current BootstrapFewShot has two different demo paths, and only one of them is metric-gated.
Conceptually:
2. Compile a few-shot candidate
Using the Chapter 7 splits and Chapter 9 metric:
After compilation, evaluate the candidate with the same development harness:
3. Inspect what changed
The candidate is not mystical. It has different program state.
fewshot.save("artifacts/editorial_fewshot_candidate.json")
You can inspect the current predictor state directly:
4. Labeled demos and bootstrapped demos
A labeled demo and a bootstrapped demo differ in both origin and structure.
Current LabeledFewShot samples rows from the trainset and assigns them as predictor demonstrations. At formatting time, a demo is useful only to the extent that its fields overlap the input and output fields of that predictor’s signature.
That exposes a design issue in our teaching corpus:
5. Fair comparison
The comparison must hold the inference protocol steady:
Writer’s editorial data illustrates why this matters. Accepted author edits can be valuable labels for the behavior we eventually want to teach, but they are not automatically demonstrations for every internal predictor in a composed program. They first need to be transformed into examples whose fields and provenance match the task being optimized.
Historical edits can also encode workflow noise: domain mixing, inconsistent reviewer preference, synthetic or derived pairs, stale context, or decisions made under a different model and interface. Filtering and deterministic split assignment make the experiment easier to interpret, but neither step replaces inspection of the examples that actually became candidate program state.
6. Failure modes
Few-shot optimization often fails for ordinary data reasons.
| Failure | What it looks like |
|---|---|
| weak bootstrap metric | Low-quality traces become augmented demonstrations |
| label/signature mismatch | max_labeled_demos is nonzero but training rows do not demonstrate leaf outputs |
| bad stage labels | The program imitates incorrect intermediate analysis or assessment |
| duplicate source families | Dev/holdout behavior is indirectly represented in optimizer-visible data |
| overly specific demos | The program copies style from one chapter or domain |
| context pressure | Demonstrations consume tokens needed for the actual task |
| weak provenance | Saved demos cannot be traced back to source cases or admission scores |
| accidental style | The model learns “always compress” or “always dramatize” |
The cure is rarely “try more optimizer settings” first. Inspect the actual predictor demos, the source cases, the threshold decisions, and the metric.
What Usually Goes Wrong
| Symptom | Likely cause | How to diagnose it | What to change |
|---|---|---|---|
max_labeled_demos is nonzero but useful labeled demos are absent |
Train rows do not contain fields matching the leaf predictor outputs | Compare demo keys with each predictor Signature | Create stage-specific labeled examples or disable labeled demos |
| Training score rises, dev score falls | Demonstrations overfit or exploit the metric | Compare train/dev failures and selected traces | Clean examples, strengthen the metric, or reduce demo pressure |
| Candidate copies one author’s rhythm everywhere | Demo set is stylistically narrow | Read predictor demos and source provenance | Split by domain or add diversity controls |
| A demo cannot be traced to its source case | Case lineage was not recorded during bootstrapping | Compare saved demos with the optimization audit | Persist source case ID, predictor name, score, and demo fingerprint during compile |
| Compile consumes more calls than expected | Bootstrap attempts or extra rounds multiply LM executions | Record max_rounds, accepted attempts, LM usage, and teacher settings |
Reduce rounds or budget only after inspecting where calls were spent |
| Candidate cannot fit the target context budget | Demonstrations consume too much prompt space | Inspect rendered history/prompt usage under the target LM | Use fewer or smaller demos |
| Holdout behavior appears in demos | Source-level leakage or near-duplicates crossed the split | Compare source groups and fingerprints, not only demo IDs | Rebuild the split and rerun the entire experiment |
Conclusion
We gained a mechanism for changing predictor behavior through demonstrations constructed from metric-approved traces and, when the data is signature-compatible, labeled training examples.
We removed the assumption that “few-shot data” is one undifferentiated collection of accepted rewrites. Demonstrations belong to particular predictor signatures, and their source, admission rule, and lineage are part of the candidate program’s provenance.
The remaining limitation is that few-shot optimization mainly changes demonstration state. The instructions themselves are still largely fixed. Chapter 12 lets the optimizer search that part of the program as well.
