From Experiment to Production

Chapter 17 gave us an experimental firewall that blocked every constructed leakage attack without blocking clean payloads. That makes the next question meaningful: once evaluation evidence is trustworthy, when does that evidence justify changing the active system?

That still does not mean:

optimizer.compile(...)
        ↓
production

The lifecycle is closer to:

candidate artifact
      ↓
independent evaluation
      ↓
comparison policy
      ↓
promotion decision
      ↓
activation
      ↓
observation
      ↓
rollback if necessary

Promotion is not an optimizer feature. It is a governance boundary.

1. Offline win is insufficient

A candidate may beat the aggregate metric and still be unacceptable:

hard regression introduced
invalid outputs increased
minority case got worse
scope constraints broke
production model differs from evaluation model
tool dependency unavailable
artifact cannot be reproduced
latency/cost unacceptable

CoCoder’s promotion policy does not collapse everything into one arbitrary weighted score. It compares dimensions: completed executions, invalid output, scope rejection, validation pass/fail, motivating concern status, material regressions, and duration. It rejects candidates that increase hard regressions or reduce validation passes.

That is the right lesson for DSPy programs. A better optimizer score is evidence. It is not activation.

The Chapter 17 governance experiment makes this concrete. One synthetic candidate scored 0.95, far above the active program’s 0.72, but introduced a hard failure. The policy returned REJECT. Another 0.95 candidate introduced a validation regression and was also rejected.

Aggregate score did not override hard constraints.

2. CoCoder’s promotion boundary

CoCoder stores separate records:

ProgramEvaluationCaseResultDTO
ProgramEvaluationDTO
ProgramComparisonDTO
ProgramPromotionDTO

The docs summarize the invariant:

optimizer proposes
CoCoder evaluates
CoCoder compares
operator/system policy promotes

Promotion requires a persisted comparison, a promote recommendation unless there is an explicit override, the baseline still being active, the candidate still being a candidate, and optimizer lineage matching the active baseline.

The deterministic Chapter 17 policy exercised seven scenarios:

The distinction between the last two decision classes matters. REJECT means the available evidence contains a disqualifying regression. INSUFFICIENT means the available evidence does not justify activation.

Transactional activation looks like:

v1 active      -> v1 superseded
v2 candidate   -> v2 active

Rejection does not delete the candidate. Rollback restores a historical active/superseded version and records decision="rolled_back".

3. DSPy artifact vs deployment manifest

DSPy can save compiled program state:

candidate.save("artifacts/editorial_candidate.json")

loaded = EditorialRewriteProgram()
loaded.load(path="artifacts/editorial_candidate.json")

That file is not the deployment record.

The deployment manifest still needs:

program id/version
DSPy saved-state path/fingerprint
model/provider/config
tool surface version
retrieval/index version
metric/evaluation evidence
promotion decision
operator
activation time
rollback target

Without this, you may have a compiled program but no way to explain why it became active.

The measured state-machine run recorded separate fingerprints for:

• the candidate artifact;
• the promotion decision;
• the deployment manifest;
• the activation record;
• the rollback record.

The promotable candidate artifact fingerprint was 6327fd1c...f7cc8. Its promotion decision, active deployment manifest, and activation record all had different fingerprints.

That is deliberate. A compiled program is not a decision, a decision is not a deployment, and a deployment is not an activation event.

4. Service boundary

A minimal service shape:

HTTP/API/request
       ↓
input validation
       ↓
active program registry
       ↓
DSPy program
       ↓
tool/runtime dependencies
       ↓
output validation
       ↓
run record

DSPy docs discuss deployment approaches including ordinary service frameworks and observability integrations such as MLflow. Those are useful mechanisms, not the architecture itself. A project can also maintain its own event layer, as Writer does with candidate provenance and preference/judge observations.

5. Observability is part of activation

A production run should produce a record even when nothing dramatic happens.

def production_run_record(request_id: str, program_version: str, output: dict) -> dict:
    return {
        "request_id": request_id,
        "program_id": "editorial_rewrite_program",
        "program_version": program_version,
        "model_provider": output.get("model_provider"),
        "model_name": output.get("model_name"),
        "fallback_state": output.get("fallback_state"),
        "validation_status": output.get("validation_status"),
        "failure_reason": output.get("failure_reason"),
        "input_fingerprint": output.get("input_fingerprint"),
        "output_fingerprint": output.get("output_fingerprint"),
        "trace_id": output.get("trace_id"),
    }

For repository repair, add tool calls, changed files, test targets, validation artifact IDs, scope result, and rollback eligibility. For editorial work, add candidate source, evidence packet hash, review decision, application status, and later outcome evidence.

The record is not only for dashboards. It becomes the raw material for the next controlled dataset after filtering and audit.

Compounding Engineering is useful as a real-system reference because it emphasizes codifying learning from completed engineering work and injecting relevant knowledge into future work. This book adds the stricter experimental boundary: production observation is not immediately valid training knowledge. The outcome has to mature, retain provenance, pass leakage policy, and be represented in decision-time form before it becomes future evidence. Source: Strategic-Automation/dspy-compounding-engineering.

6. Shadow, canary, activation

Safe activation is staged.

Shadow:

request
  |
  +--> active program --> user
  |
  +--> candidate ------> recorded only

Canary:

small bounded subset
      ↓
candidate controls result
      ↓
monitor hard failures

Full activation happens only after acceptable evidence. DSPy does not automatically provide that policy. The surrounding system owns it.

The Chapter 17 runner enforced this boundary directly. editorial-rewrite-v2 became active only after the PROMOTE scenario. The runner then attempted to activate a candidate whose comparison decision was REJECT; activation raised activation attempted without PROMOTE decision.

Promotion was therefore not merely advisory metadata. It was an executable gate.

7. Production feedback is not automatically a label

Be careful:

program produced result
        ↓
user did not complain

≠

positive label

Also:

human preferred candidate
        ↓
candidate was not applied

≠

successful production outcome

Writer’s architecture distinguishes preference evidence, candidate generation evidence, blind review, application evidence, and later statistics/ranker evaluation. That separation keeps the next dataset cleaner.

8. Rollback is a feature, not a failure

If a system can promote, it must be able to reverse promotion:

Rollback is not only a deployment command. It is a rehearsed reconstruction of the last known acceptable program boundary. If an old candidate state loads but its provider, retrieval index, tool schema, or validation dependency has drifted, rollback may only restore the appearance of the old system.

The experiment rehearsed the state transition rather than merely naming a rollback target:

editorial-rewrite-v1
    eligible
      ↓
editorial-rewrite-v2
    candidate
      ↓ PROMOTE
editorial-rewrite-v2
    active
      ↓ simulated hard regression
editorial-rewrite-v1
    restored active

The rollback logic selected the previous eligible deployment and produced a separate rollback record with its own fingerprint.

active v4
   ↓
problem detected
   ↓
reactivate historical v3
   ↓
record rollback

CoCoder’s rollback operation requires a target version that was historically active or superseded, changes the current active version to rolled_back, restores the target to active, and persists a promotion record with decision="rolled_back". That makes rollback part of lineage rather than an invisible database edit.

What Usually Goes Wrong

Conclusion

We gained a production lifecycle for DSPy programs and exercised its decision boundary.

The deterministic governance run produced all three comparison outcomes: one PROMOTE, two REJECT, and four INSUFFICIENT. A candidate scoring 0.95 was still rejected when it introduced either a hard failure or a validation regression. Protocol mismatch, stale baseline lineage, too few cases, and sub-threshold improvement did not prove the candidate bad; they produced INSUFFICIENT because the evidence did not justify activation.

The successful candidate moved through separate artifacts:

candidate artifact
    ↓
evaluation
    ↓
promotion decision
    ↓
deployment manifest
    ↓
activation record

A rejected candidate could not cross the activation gate. After activation, rollback restored the previous eligible editorial-rewrite-v1 deployment and recorded the reversal separately.

We removed the assumption that an optimized program should serve real work automatically.

All mechanisms are now available separately. The final chapter combines them into one engineering program.