Build a Self-Improving Engineering Program

We can now assemble the book.

The capstone experiment executes the complete repository-repair loop rather than leaving the language-model stages hypothetical. A local Qwen3-backed program inspected a bounded repository, diagnosed a concrete defect, proposed a repair, passed deterministic development validation, passed an independent hidden evaluation, and reached a separate PROMOTE decision.

The result matters because each stage remained independently inspectable.

The target is not an autonomous system that rewrites itself and deploys the result. That is the weak meaning of self-improvement.

The stronger meaning is:

The system creates an evidence-generating loop in which program behavior can be improved empirically under explicit experimental and promotion boundaries.

The capstone task is repository repair.

Input:

repository
issue / requested engineering change
constraints

Output:

diagnosis
evidence
proposed intervention
candidate patch
validation result
confidence / uncertainty

The real success criterion is external:

Did the change resolve the motivating problem
without causing unacceptable regressions?

1. The insufficient baseline

The smallest DSPy version is tempting:

import dspy

repair = dspy.Predict("issue, repository_context -> diagnosis, patch")

It fails for reasons the book has already exposed:

contract too vague
context too large
no tool boundary
no validation loop
no split discipline
no promotion boundary
patch similarity confused with success

So we build the real program from parts.

In the measured fixture, the baseline invoice_total implementation ignored quantity. The public test failed, and the independent hidden evaluation scored the broken baseline at 0.5. A known deterministic control patch passed both public and hidden validation, proving that the harness could recognize the intended repair before the LM path was evaluated.

2. Capstone architecture

                        issue
                          |
                          v
                  normalize request
                          |
                          v
                 repository explorer
                /         |         \
          search code   read files   inspect tests
                \         |         /
                 \        |        /
                  evidence packet
                          |
                          v
                       diagnose
                          |
                          v
                  choose intervention
                          |
                          v
                    generate patch
                          |
                          v
                isolated application
                          |
                          v
                     validation
                    /          \
                  pass         fail
                   |             |
                   |          interpret
                   |             |
                   |           revise
                   |             |
                   +------<------+
                          |
                          v
                    final candidate

LM responsibilities:

interpret issue
choose evidence to inspect
diagnose likely cause
propose intervention
generate patch
interpret validation failure

Deterministic responsibilities:

repository revision
file access
scope checks
patch application
worktree isolation
test execution
schema validation
hashing
artifact persistence
comparison
promotion state

CoCoder’s current candidate generation program embodies this separation. Its program definition specifies input schema, output schema, constraints, runtime policy, validation contract, optimization surface, and a direct runtime. The evaluation adapter prepares an isolated workspace, builds repository analysis context, materializes candidate generation, runs scope and validation, analyzes the candidate, computes engineering delta, and finalizes the report.

The capstone follows the same division. The LM received only three read-only tools—repository search, bounded file reads, and symbol inspection. It had no mutation, shell, network, or hidden-validation capability. It made three diagnosis tool calls and identified service.py as the implicated file.

3. Program contracts

class DiagnoseIssue(dspy.Signature):
    """Diagnose a repository issue from bounded evidence."""

    issue: str = dspy.InputField()
    repository_evidence: str = dspy.InputField()
    constraints: str = dspy.InputField()

    diagnosis: str = dspy.OutputField()
    suspected_files: list[str] = dspy.OutputField()
    missing_evidence: str = dspy.OutputField()
    confidence: float = dspy.OutputField()

class GeneratePatch(dspy.Signature):
    """Generate a bounded unified diff for a diagnosed issue."""

    issue: str = dspy.InputField()
    diagnosis: str = dspy.InputField()
    repository_evidence: str = dspy.InputField()
    constraints: str = dspy.InputField()

    patch: str = dspy.OutputField(desc="Unified diff or explicit rejection")
    rationale: str = dspy.OutputField()
    expected_behavior_change: str = dspy.OutputField()

class InterpretValidationFailure(dspy.Signature):
    """Explain a failed patch validation and propose the next revision strategy."""

    issue: str = dspy.InputField()
    diagnosis: str = dspy.InputField()
    patch: str = dspy.InputField()
    validation_output: str = dspy.InputField()

    failure_analysis: str = dspy.OutputField()
    revision_strategy: str = dspy.OutputField()

The contract does not include promotion_decision, historical validation outcome, or the gold patch. Those belong to evaluation and governance.

4. Validation loop

candidate patch
      ↓
apply in isolated worktree
      ↓
parse / compile / lint where relevant
      ↓
focused tests
      ↓
broader tests
      ↓
scope/regression checks
      ↓
result

An LM judge can help interpret confusing failures. It does not replace tests. The patch must still apply. The target behavior must still be checked. Scope must still be controlled.

In the measured run, the first generated repair applied successfully, parsed as Python, modified only service.py, and passed the public development test. Because development validation passed, no revision was allowed or needed.

Only then did the independent hidden evaluator run. The candidate passed all hidden cases for an aggregate score of 1.0. That hidden result was never fed back into patch generation.

CoCoder records dimensions such as parse/schema status, scope status, validation status, motivating-concern status, material regression count, and limitations. That is the shape to emulate.

5. Optimization dataset

A repository-repair case should record:

case_id
repository revision
issue
allowed evidence
constraints
baseline program version
candidate patch
execution status
validation status
scope status
concern resolution
regressions
human decision if available
split

Only decision-time fields enter program inputs. Outcome fields support metrics and promotion.

The capstone enforces that rule operationally. Public development-test output may guide at most one repair revision. Independent hidden-evaluation results are unavailable to diagnosis and patch generation and become visible only after the final candidate has passed development validation.

That creates a one-way evidence boundary:

diagnosis / repair
    ↓
development validation
    ↓
optional bounded revision
    ↓
final candidate
    ↓
independent evaluation
    ↓
comparison / promotion

Evidence below the final-candidate boundary cannot flow upward into repair.

The metric should prefer outcomes:

patch valid
scope acceptable
validation passes
motivating concern resolved
no material regression

Patch similarity to a gold diff may be a small diagnostic signal. It is not the success criterion unless the task is specifically “reproduce this patch.”

6. Choose the optimizer by failure

Do not stack optimizers for decoration.

For a repository repair program, GEPA becomes attractive when validation failures can produce actionable feedback:

The patch edited the target file but did not update the caller that still passes the old argument.

That feedback can guide instruction mutation better than a scalar zero.

7. Frozen experiment and promotion

The manifest:

program version
repository case revisions
dataset fingerprint
train/dev/holdout IDs
runtime version
provider/model config
tool surface version
retrieval/index version
validation contract
metric version
optimizer version/config
DSPy version
seed
budget
source revision

The protocol:

freeze
  ↓
baseline evaluation
  ↓
optimization on allowed evidence
  ↓
candidate persistence
  ↓
integrity audit
  ↓
holdout evaluation
  ↓
comparison
  ↓
promotion recommendation

Activation:

baseline active v1
       ↓
candidate v2
       ↓
holdout evaluation
       ↓
comparison policy
       ↓
PROMOTE / REJECT / INSUFFICIENT
       ↓
operator decision
       ↓
activation

Rollback remains available.

In the capstone run, the candidate artifact and promotion decision were separate persisted objects with different fingerprints. The candidate artifact fingerprint was 2840b5d5...b4c85; the promotion decision fingerprint was 60df345c...eedc5.

The decision was PROMOTE because the final candidate passed development validation, scored 1.0 on the independent evaluation, and introduced no hard regression. Activation was deliberately not performed automatically.

That preserves the Chapter 17 invariant: promotion evidence can justify activation without itself being activation.

8. A small executable manifest guard

The capstone needs ordinary tests around the experiment boundary.

def validate_experiment_manifest(manifest: dict) -> None:
    required = {
        "program_version",
        "dataset_fingerprint",
        "train_ids",
        "dev_ids",
        "holdout_ids",
        "runtime_version",
        "provider_config",
        "tool_surface_version",
        "validation_contract",
        "metric_version",
        "optimizer_config",
        "dspy_version",
        "seed",
        "budget",
    }
    missing = sorted(required - set(manifest))
    if missing:
        raise ValueError(f"manifest missing: {', '.join(missing)}")

    visible = set(manifest["train_ids"]) | set(manifest["dev_ids"])
    leaked = visible & set(manifest["holdout_ids"])
    if leaked:
        raise ValueError(f"holdout ids optimizer-visible: {', '.join(sorted(leaked))}")

This is not a substitute for CoCoder’s full experiment service. It is the smallest version of the same contract: the candidate cannot be interpreted if the manifest is incomplete or the holdout boundary is broken.

The measured capstone manifest passed its executable guard with no missing fields and no forbidden fields. It recorded the provider, tool surface, development-validation contract, independent-evaluation protocol, candidate artifact, promotion decision, DSPy version, repair budget, and source revision.

The manifest therefore explains not only what candidate existed, but under which evidence and governance boundary it was produced.

9. Production evidence loop

The finished loop:

active program
      ↓
real tasks
      ↓
run evidence
      ↓
validation
      ↓
human decisions/outcomes
      ↓
audited dataset
      ↓
next experiment
      ↓
candidate program
      ↓
independent evaluation
      ↓
promotion decision

This is the book’s final movement:

prompt engineering
        ↓
language-model programming
        ↓
empirical program engineering

What Usually Goes Wrong

The capstone fixture in experiments/dspy-from-first-principles/ch18_repository_repair starts with a concrete defect: invoice totals ignore quantity.

The measured execution completed the full path with the canonical local model:

The successful first repair is useful, but it is not the experiment’s main proof. The same harness would remain valid if the LM failed, because repair quality was deliberately not made an invariant. The deterministic control, bounded tool surface, validation boundary, independent evaluation, manifest guard, and promotion record determine whether the result is interpretable.

Compounding Engineering and Arachne provide useful contrasts: retained learning across work and graph execution/healing, respectively. Sources: Strategic-Automation/dspy-compounding-engineering, Strategic-Automation/arachne.

Conclusion

We did not build a prompt optimizer. We built the architecture around a language-model program.

The capstone then exercised that architecture. A language model diagnosed a real fixture defect and proposed a valid repair on its first attempt. But the model was never allowed to certify its own success. Ordinary software applied the patch, checked syntax and scope, ran development validation, withheld independent evaluation until the candidate was frozen, compared the resulting evidence, audited the manifest, and recorded a separate promotion decision.

The completed system has a contract, execution strategy, tools, context policy, examples, metric, optimizer, experiment manifest, candidate artifact, independent evidence, promotion policy, production telemetry, and rollback.

Across the book, the unit of progress changed:

prompt wording
    ↓
program structure
    ↓
measured behavior
    ↓
controlled optimization
    ↓
independent evidence
    ↓
governed deployment
    ↓
new audited evidence

That is the stronger meaning of self-improvement. The system does not silently rewrite itself and declare success. It repeatedly creates candidates whose behavior can be challenged by evidence outside the mechanism that produced them.

The final thesis is simple:

A language-model application becomes engineerable when its behavior can be specified, executed, observed, evaluated, diagnosed, optimized, versioned, compared, promoted, and improved again under evidence.