# Build a Self-Improving Engineering Program
# Book: DSPy From First Principles
# Chapter URL: https://programmer.ie/books/dspy-from-first-principles/20-chapter/
# Prose fingerprint (sha256, normalized): 887718db9a87ab11b438a2eba15e4c136ac110f30ac5c89fe8703a11b5f0a7bc
# Snapshot generated at build time by Hugo. This is the complete rendered prose
# of the chapter; it is not a transcript of the source file.

We can now assemble the book.
Every prior chapter added one mechanism because the system before it had exposed a specific limitation: a contract (Chapter 3), a measured baseline (Chapter 8), an attacked metric (Chapter 9), three optimizers and an objective swap (Chapters 11–14), tools and bounded environment access (Chapter 15), evidence selection (Chapter 16), reasoning-path search (Chapter 17), an experimental firewall (Chapter 18), and a promotion boundary with rollback (Chapter 19). This chapter does not introduce another mechanism. It asks whether the ones we have compose into a single loop without any stage quietly promoting its own output.
The capstone experiment executes the complete repository-repair loop rather than leaving the language-model stages hypothetical. A local Qwen3-backed program inspected a bounded repository, diagnosed a concrete defect, proposed a repair, passed deterministic development validation, passed an independent hidden evaluation, and reached a separate PROMOTE decision.
The result matters because each stage remained independently inspectable.
The target is not an autonomous system that rewrites itself and deploys the result. That is the weak meaning of self-improvement.
The stronger meaning is:
The system creates an evidence-generating loop in which program behavior can be improved empirically under explicit experimental and promotion boundaries.
The capstone task is repository repair.
Input:
repository issue / requested engineering change constraints Output:
diagnosis evidence proposed intervention candidate patch validation result confidence / uncertainty The real success criterion is external:
Did the change resolve the motivating problem without causing unacceptable regressions? 1. The insufficient baseline The smallest DSPy version is tempting:
import dspy repair = dspy.Predict("issue, repository_context -> diagnosis, patch") It fails for reasons the book has already exposed:
contract too vague context too large no tool boundary no validation loop no split discipline no promotion boundary patch similarity confused with success So we build the real program from parts.
In the measured fixture, the baseline invoice_total implementation ignored quantity. The public test failed, and the independent hidden evaluation scored the broken baseline at 0.5. A known deterministic control patch passed both public and hidden validation, proving that the harness could recognize the intended repair before the LM path was evaluated.
2. Capstone architecture flowchart TD I[issue] --> N[normalize request] N --> RE[repository explorer] RE --> SC[search code] RE --> RF[read files] RE --> IT[inspect tests] SC --> EP[evidence packet] RF --> EP IT --> EP EP --> DG[diagnose] DG --> CI[choose intervention] CI --> GP[generate patch] GP --> IA[isolated application] IA --> VA{validation} VA -->|pass| FC[final candidate] VA -->|fail| RV[interpret and revise, at most once] RV --> IA This is the loop that actually ran. Everything left of final candidate is the LM-and-tools path; everything from validation rightward is deterministic. The fail branch is bounded — one revision, then the candidate is what it is.
LM responsibilities:
interpret issue choose evidence to inspect diagnose likely cause propose intervention generate patch interpret validation failure Deterministic responsibilities:
repository revision file access scope checks patch application worktree isolation test execution schema validation hashing artifact persistence comparison promotion state CoCoder’s current candidate generation program embodies this separation. Its program definition specifies input schema, output schema, constraints, runtime policy, validation contract, optimization surface, and a direct runtime. The evaluation adapter prepares an isolated workspace, builds repository analysis context, materializes candidate generation, runs scope and validation, analyzes the candidate, computes engineering delta, and finalizes the report.
The capstone follows the same division. The LM received only three read-only tools—repository search, bounded file reads, and symbol inspection. It had no mutation, shell, network, or hidden-validation capability. It made three diagnosis tool calls and identified service.py as the implicated file.
3. Program contracts class DiagnoseIssue(dspy.Signature): """Diagnose a repository issue from bounded evidence.""" issue: str = dspy.InputField() repository_evidence: str = dspy.InputField() constraints: str = dspy.InputField() diagnosis: str = dspy.OutputField() suspected_files: list[str] = dspy.OutputField() missing_evidence: str = dspy.OutputField() confidence: float = dspy.OutputField() class GeneratePatch(dspy.Signature): """Generate a bounded unified diff for a diagnosed issue.""" issue: str = dspy.InputField() diagnosis: str = dspy.InputField() repository_evidence: str = dspy.InputField() constraints: str = dspy.InputField() patch: str = dspy.OutputField(desc="Unified diff or explicit rejection") rationale: str = dspy.OutputField() expected_behavior_change: str = dspy.OutputField() class InterpretValidationFailure(dspy.Signature): """Explain a failed patch validation and propose the next revision strategy.""" issue: str = dspy.InputField() diagnosis: str = dspy.InputField() patch: str = dspy.InputField() validation_output: str = dspy.InputField() failure_analysis: str = dspy.OutputField() revision_strategy: str = dspy.OutputField() The contract does not include promotion_decision, historical validation outcome, or the gold patch. Those belong to evaluation and governance.
4. Validation loop The validation node in section 2 expands into a fixed sequence: apply the candidate patch in an isolated worktree; parse, compile, and lint where relevant; run the focused tests, then the broader tests; then the scope and regression checks. Any step failing produces the result — there is no partial credit for a patch that applies but changes the wrong files.
An LM judge can help interpret confusing failures. It does not replace tests. The patch must still apply. The target behavior must still be checked. Scope must still be controlled.
In the measured run, the first generated repair applied successfully, parsed as Python, modified only service.py, and passed the public development test. Because development validation passed, no revision was allowed or needed.
Only then did the independent hidden evaluator run. The candidate passed all hidden cases for an aggregate score of 1.0. That hidden result was never fed back into patch generation.
CoCoder records dimensions such as parse/schema status, scope status, validation status, motivating-concern status, material regression count, and limitations. That is the shape to emulate.
5. Optimization dataset A repository-repair case should record:
case_id repository revision issue allowed evidence constraints baseline program version candidate patch execution status validation status scope status concern resolution regressions human decision if available split Only decision-time fields enter program inputs. Outcome fields support metrics and promotion.
The capstone enforces that rule operationally. Public development-test output may guide at most one repair revision. Independent hidden-evaluation results are unavailable to diagnosis and patch generation and become visible only after the final candidate has passed development validation.
That creates a one-way evidence boundary:
diagnosis / repair ↓ development validation ↓ optional bounded revision ↓ final candidate ↓ independent evaluation ↓ comparison / promotion Evidence below the final-candidate boundary cannot flow upward into repair.
The metric should prefer outcomes:
patch valid scope acceptable validation passes motivating concern resolved no material regression Patch similarity to a gold diff may be a small diagnostic signal. It is not the success criterion unless the task is specifically “reproduce this patch.”
6. Choose the optimizer by failure Do not stack optimizers for decoration.
Failure Reasonable optimizer Few trusted repair examples exist BootstrapFewShot Instructions systematically weak MIPROv2 Failures have rich diagnostics GEPA Large context selection fails Improve retrieval/tools before prompt optimization For a repository repair program, GEPA becomes attractive when validation failures can produce actionable feedback:
The patch edited the target file but did not update the caller that still passes the old argument. That feedback can guide instruction mutation better than a scalar zero.
7. Frozen experiment and promotion The manifest:
program version repository case revisions dataset fingerprint train/dev/holdout IDs runtime version provider/model config tool surface version retrieval/index version validation contract metric version optimizer version/config DSPy version seed budget source revision The protocol:
freeze ↓ baseline evaluation ↓ optimization on allowed evidence ↓ candidate persistence ↓ integrity audit ↓ holdout evaluation ↓ comparison ↓ promotion recommendation Activation:
baseline active v1 ↓ candidate v2 ↓ holdout evaluation ↓ comparison policy ↓ PROMOTE / REJECT / INSUFFICIENT ↓ operator decision ↓ activation Rollback remains available.
In the capstone run, the candidate artifact and promotion decision were separate persisted objects with different fingerprints. The candidate artifact fingerprint was 2840b5d5...b4c85; the promotion decision fingerprint was 60df345c...eedc5.
The decision was PROMOTE because the final candidate passed development validation, scored 1.0 on the independent evaluation, and introduced no hard regression. Activation was deliberately not performed automatically.
That preserves the Chapter 19 invariant: promotion evidence can justify activation without itself being activation.
8. A small executable manifest guard The capstone needs ordinary tests around the experiment boundary.
def validate_experiment_manifest(manifest: dict) -> None: required = { "program_version", "dataset_fingerprint", "train_ids", "dev_ids", "holdout_ids", "runtime_version", "provider_config", "tool_surface_version", "validation_contract", "metric_version", "optimizer_config", "dspy_version", "seed", "budget", } missing = sorted(required - set(manifest)) if missing: raise ValueError(f"manifest missing: {', '.join(missing)}") visible = set(manifest["train_ids"]) | set(manifest["dev_ids"]) leaked = visible & set(manifest["holdout_ids"]) if leaked: raise ValueError(f"holdout ids optimizer-visible: {', '.join(sorted(leaked))}") This is not a substitute for CoCoder’s full experiment service. It is the smallest version of the same contract: the candidate cannot be interpreted if the manifest is incomplete or the holdout boundary is broken.
The measured capstone manifest passed its executable guard with no missing fields and no forbidden fields. It recorded the provider, tool surface, development-validation contract, independent-evaluation protocol, candidate artifact, promotion decision, DSPy version, repair budget, and source revision.
The manifest therefore explains not only what candidate existed, but under which evidence and governance boundary it was produced.
9. Production evidence loop The finished loop:
flowchart LR AP[active program] --> RT[real tasks] RT --> RE[run evidence] RE --> VA[validation] VA --> HD[human decisions / outcomes] HD --> AD[audited dataset] AD --> NE[next experiment] NE --> CP[candidate program] CP --> IE[independent evaluation] IE --> PD[promotion decision] PD -->|on PROMOTE| AP Production activity becomes the next experiment’s evidence only after it matures and clears the leakage policy. This is the book’s final movement:
prompt engineering ↓ language-model programming ↓ empirical program engineering What Usually Goes Wrong Symptom Likely cause How to diagnose it What to change Patch is syntactically valid but concern persists Metric only checks apply/parse Inspect motivating concern status Add behavior-focused validation Tests pass but scope changed Scope validation absent Compare changed files and engineering delta Add scope gate Optimizer score improves but candidate rejected Governance dimensions caught regression Read comparison losses Fix program, not policy Program uses historical answer Memory/tool leakage Inspect evidence packet Enforce firewall by split/revision Active behavior cannot be explained Missing manifests/run records Trace active version lineage Persist program, model, tool, and promotion evidence The capstone fixture in experiments/dspy-from-first-principles/ch18_repository_repair starts with a concrete defect: invoice totals ignore quantity.
The measured execution completed the full path. The run is identified below so the whole experiment can be inspected in one place:
Identity Value Run repair-run-20260901-013940 (experiments/dspy-from-first-principles/ch18_repository_repair) Model ollama_chat/qwen3:latest — Qwen3 8.2B, Q4_K_M, temperature=0, native_thinking=false, DSPy 3.3.1 Fixture invoice_total ignores quantity; fixture fingerprint 85b9f606…a2ff8f Tool surface repository search, bounded file read, symbol inspection (no mutation, shell, network, or hidden-validation) Candidate artifact fingerprint 2840b5d5…3b4c85 Promotion decision fingerprint 60df345c…6eedc5 Evaluation protocol fingerprint bee8ace4…800843 Experiment fingerprint 3d4483a6…242884 Stage Result Broken public validation failed Broken hidden score 0.5 Known control patch passed public + hidden Diagnosis completed Read-only diagnosis tool calls 3 First LM patch applied yes Syntax gate passed Scope gate passed Public development validation passed Revision not needed Independent hidden evaluation passed Hidden score 1.0 Promotion decision PROMOTE Automatic activation no Total LM history entries 6 Total LM tokens 7,065 Manifest guard passed The successful first repair is useful, but it is not the experiment’s main proof. The same harness would remain valid if the LM failed, because repair quality was deliberately not made an invariant. The deterministic control, bounded tool surface, validation boundary, independent evaluation, manifest guard, and promotion record determine whether the result is interpretable.
Compounding Engineering and Arachne provide useful contrasts: retained learning across work and graph execution/healing, respectively. Sources: Strategic-Automation/dspy-compounding-engineering, Strategic-Automation/arachne.
Conclusion We did not build a prompt optimizer. We built the architecture around a language-model program.
The capstone then exercised that architecture. A language model diagnosed a real fixture defect and proposed a valid repair on its first attempt. But the model was never allowed to certify its own success. Ordinary software applied the patch, checked syntax and scope, ran development validation, withheld independent evaluation until the candidate was frozen, compared the resulting evidence, audited the manifest, and recorded a separate promotion decision.
The completed system has a contract, execution strategy, tools, context policy, examples, metric, optimizer, experiment manifest, candidate artifact, independent evidence, promotion policy, production telemetry, and rollback.
Across the book, the unit of progress changed:
prompt wording ↓ program structure ↓ measured behavior ↓ controlled optimization ↓ independent evidence ↓ governed deployment ↓ new audited evidence That is the stronger meaning of self-improvement. The system does not silently rewrite itself and declare success. It repeatedly creates candidates whose behavior can be challenged by evidence outside the mechanism that produced them.
The final thesis is simple:
A language-model application becomes engineerable when its behavior can be specified, executed, observed, evaluated, diagnosed, optimized, versioned, compared, promoted, and improved again under evidence.