The Metric Is Part of the Program
The Metric Is Part of the Program
Chapter 8 gave us a baseline harness. It deliberately used a weak smoke-test metric because the first job was to make evaluation run.
Now the dangerous part begins.
During optimization, DSPy searches for program states that score well under the metric we provide. The optimizer does not know whether the score is a faithful proxy for the behavior we actually want.
1. The metric ladder
Different tasks tolerate different metrics.
| Metric type | Useful when | Weakness |
|---|---|---|
| exact match | One canonical output is genuinely required | Rejects valid alternatives when answers are not unique |
| deterministic invariants | A failure property is directly computable | Usually measures only part of quality |
| reference comparison | A reference is a useful target or anchor | Penalizes valid alternatives and can reward imitation |
| semantic similarity | Meaning preservation matters | Can reward generic or stylistically weak paraphrases |
| LM judge | Qualitative judgment is difficult to encode directly | Bias, drift, cost, and judge calibration |
| human preference | Human choice is itself an important signal | Slow, sparse, noisy, and not identical to downstream outcome |
| composite metric | Several dimensions must be represented together | Weights and interactions create new attack surfaces |
The editorial task needs a composite metric. Some failures are hard constraints. Others are soft quality judgments.
2. Hard constraints first
Hard constraints should be deterministic when possible.
3. Reference overlap is only a signal
A simple token-set overlap signal can measure lexical closeness to a reference, but it should not dominate.
4. A composite teaching metric
Here is a model-independent metric we can use in Chapters 10-12. It is intentionally conservative and incomplete.
5. Attack the metric
A metric should be tested against adversarial candidates before it is exposed to an optimizer.
Do not use the holdout to design those attacks. The entity-bearing edit-002 training case is sufficient:
6. LM judges need provenance
Writer’s contextual preference judge is a useful architecture, but it is not magic. It uses a DSPy signature to judge original versus candidate in context, asks for a structured decision and reason codes, parses the result, applies deterministic hard policy, and records model/provider/config/input fingerprints.
For an LM judge, provenance should identify the judge separately from the program being judged: judge signature/prompt version, provider, model, configuration, input fingerprint, and any ordering or rubric choices that can affect the verdict.
That is the right shape:
deterministic guardrails
↓
LM judge
↓
parsed structured result
↓
human/evaluation comparison
↓
recorded provenance
LM-as-judge risks include correlated errors between task model and judge, verbosity bias, position bias, rubric sensitivity, calibration drift, and cost. A judge’s confidence is still model output. The judge itself needs evaluation against trusted labels or human decisions where possible.
Human preference and downstream outcome should also remain separate signals. A reviewer can prefer an edit that later performs poorly in its actual workflow, just as a technically valid patch can still fail to resolve the motivating issue.
For Chapters 10-12 we will use the deterministic composite metric so the examples remain runnable without a judge model. A production metric might combine deterministic gates, calibrated judge signals, human preference, and observed outcome evidence, while keeping those components individually inspectable.
What Usually Goes Wrong
| Symptom | Likely cause | How to diagnose it | What to change |
|---|---|---|---|
| Optimizer score rises but outputs get worse | Metric rewards an exploitable proxy | Read the highest-scoring failures | Add or revise checks, components, or weights and version the metric |
| Copying the original scores well | Metric rewards reference overlap without enough task-improvement signal | Run the no-edit baseline | Penalize no-op only when the case truly requires an edit |
| Entity renames pass | Soft similarity is overriding a required invariant | Run entity-rename attacks on non-holdout fixtures | Make required-entity preservation a gate |
| Appended prose still scores well | Scope penalty is too weak | Attack with reference-plus-junk candidates | Add a tested scope validator or stronger penalty |
| LM judge prefers longer answers | Judge has verbosity or presentation bias | Compare controlled pairs and candidate orderings | Calibrate the judge or add independent scope checks |
| Metric crashes during optimization | It assumes perfect prediction shape | Fuzz malformed predictions | Use defensive access and an explicit failure score |
| Metric improves after every holdout inspection | Holdout is being used for metric development | Audit the sequence of metric changes | Retire that holdout and freeze a fresh one |
Conclusion
We gained a versionable teaching objective that can drive the next optimization chapters. Its components are inspectable, its weaknesses are explicit, and its attack suite is part of the experiment rather than an afterthought.
We removed the assumption that a reference rewrite is the same thing as editorial quality, and we added a second rule: a metric is not trustworthy merely because it is deterministic.
We now have a program, data, a baseline harness, and an optimizer-facing objective. The next problem is how to let DSPy propose a candidate program without confusing optimization success with independent evidence of improvement.
