← Jev From First Principles

Appendix — Book Quality Lab: A Pi Agent with a Judgment Kernel

Run a read-only manuscript evaluator, inspect exact source evidence, and separate Pi analysis, typed judgment, policy and your editorial decision.

You can ask a language model whether a chapter is good. You then have an opinion, usually expressed with more certainty than its supporting evidence deserves. Book Quality Lab gives you something narrower and inspectable: a candidate concern, exact passages, a bounded question, a provider answer, a policy outcome and a place to record your own decision.

This appendix is an executable supplement. It preserves the verdict in Programming with Decisions. The offline implementation works, real-model inference did execute through the configured provider, and the comparative experiment returned a valid negative result: the typed judgment layer did not demonstrate superior accuracy. That finding is part of the result, not a reason to discount it.

The central conclusion of this capstone is:

A typed decision architecture does not automatically make an AI’s judgments better. Its demonstrated value here is in making decisions explicit, validated, traceable, recoverable and independently reviewable. Whether those decisions are more accurate remains an empirical question.

Run the finished offline path

From your Jev checkout, use the isolated examples package:

cd examples/book-quality-lab
npm ci --ignore-scripts
npm run setup
npm test
node cli.mjs evaluate fixtures/book --whole-book --provider rules --analysis pi-fixture

The setup command obtains the pinned μ source checkout; it does not download a model. With dependencies and source present, this pilot runs offline. The package README also shows how to select an existing μ checkout instead.

Pi executes its actual agent loop with a scripted provider. Its snapshot tool reads real fixture files through the host’s immutable inventory. Host rules prepare comparison candidates. μ’s actual decision engine validates and records the rule provider’s answers. This is observed software behavior, not evidence that an AI can edit or judge an unfamiliar book well.

The recorded acceptance suite passed 42 tests. It checks strict contract validation, wrong spans, changed sources, unavailable judges, shadow behavior, provider substitution, partial failures, human labels, read-only active mode, checkpoint resume, invalid decision targets and the real-model contract probes. A synthetic HTTP server also exercises the local adapter’s response, truncation and redirect handling. Those transport tests are not model observations. The historical measurement record is results/ch36.jsonl, metric = software_tests; the 02B diagnostic snapshots live in separate results/ch36-capstone02b-*.jsonl files and never replace it.

You can use the mechanical evaluator without μ or a model:

node cli.mjs evaluate C:/books/jev --chapter 01
node cli.mjs evaluate C:/books/jev --whole-book
node cli.mjs evaluate C:/books/pi --whole-book

You may instead supply the specific content/books/<book> directory. The CLI discovers its repository root, chapter files, concepts companions and chapter metadata. A whole-book run includes supporting Markdown pages, but does not impose numbered-chapter ordering rules on those pages.

Follow one finding through the boundaries

The fixture manuscript says:

Acceptance means permission to publish.

Its concepts companion says:

Acceptance means not permission to publish.

Both quotations are exact source lines in the recorded fixture pilot. A concepts companion is another source to compare, not an authority that wins automatically. You need both locations before you can assess the discrepancy.

    flowchart TD
  A[Manuscript and companions] --> B[Host snapshot and mechanical checks]
  B --> C[Pi reads and proposes candidates]
  B --> D[Rule-based comparison candidates]
  C --> E[Host verifies exact evidence]
  D --> E
  E --> F[Bounded typed decision in mu]
  F --> G[Provider answer and original ledger]
  G --> H[Shadow fallback or active categorization]
  H --> I[Report and human review ledger]
  

The specification asks whether these passages use the concept compatibly. The provider in the offline pilot recognizes only a narrow same-scope definition-and-negation template. It is a rule, not a model. The runtime is μ’s decision engine. The evidence includes exact excerpts, line ranges, original-byte hashes and the returned answer. The action is to retain a review item. No layer receives permission to rewrite your manuscript.

The code is in examples/book-quality-lab: manuscript.mjs owns inventory, parsing and evidence checks; analysis.mjs injects Pi’s snapshot tools; kernel.mjs defines bounded questions; evaluate.mjs records results and reports; cli.mjs exposes the workflow. The 02B contracts are contracts-v2.mjs (strict versioned schemas), context-v2.mjs (exact source contexts), structured.mjs (validated tool submission) and quality-v2.mjs (bounded review plus quality.source_findings version 3, which explicitly targets the original manuscript assertion, never the reviewer critique). SCHEMA.md specifies the artifacts.

The execution pattern connects to Pi Agents: Pi Patch Desk: the agent performs bounded work, the host owns verification and the human owns acceptance. This capstone does not replace Pi Patch Desk or change the Pi book’s dependency pin.

Inspect the decision before trusting the report

The implementation imports defineDecision, DecisionEngine, Judge and the ledger contract directly from the pinned μ source. The private package is not assumed to exist on the public npm registry. The source pin and imported-file hashes are checked before runtime import. No μ application fork is embedded in this book.

The recorded pilot ledger contains this projection of a decision:

{
  "specId": "quality.definition_consistency",
  "specVersion": 1,
  "mode": "shadow",
  "providerId": "rules:book-quality-v1",
  "modelId": "rules-v1",
  "outcome": {"status": "NEEDS_REVIEW"},
  "source": "fallback",
  "reason": "shadow",
  "judged": {"verdict": "potential_conflict"},
  "answers": {"verdict": {"type": "choice", "choice": "potential_conflict"}}
}

You can inspect the full record in evidence/book-quality-lab/pilot-release/mu-ledger.jsonl. It includes the state, question, exact passages, digest and original ledger identity. The wrapper in judgments.jsonl links that identity to its candidate finding.

Notice the difference between judged and outcome. Shadow records the rule’s conflict answer while retaining the baseline review policy. Active mode may categorize a finding using the answer, but still has no edit operation and never deletes a finding. An abstention or failure remains unresolved.

There is no reported probability in this answer. The wrapper records null, not an invented confidence. Ordinal clarity answers are positions on a rubric, not calibrated probabilities. The supplied-source claim statuses describe support, contradiction or unresolved verification; they never certify external truth.

The audit found two integration details worth preserving. μ’s current LLM adapter supplies choice probabilities when a response omits them, so this capstone instead uses a strict local completion adapter. μ’s Judge initializes unreported usage counters to zero; the original ledger is preserved, while the wrapper marks missing provider usage UNKNOWN for cost accounting. These are source-inspected implementation behaviors, not Jev API guarantees. See research/ch36-additions.md for the reuse map and license attribution.

Read the measured results at their actual scope

Policy correctness

The 02B experiment reproduced a real policy/code mismatch: the written refund runbook permitted a request that predicate_decide declined. A 108-case boundary grid showed 6 mismatches before the repair and 0 after (results/ch36-capstone02b-policy.jsonl, metric = policy_contract_repair). The owning Chapter 1 implementation was corrected; historical bytes (tickets-before.py) and the original Chapter 1 measurements are preserved. The regression is examples/ch01-generate-vs-decide/test_policy_contract.py (3 tests, 108 subtests). Scope is the literal documented policy, not a claim about correct real-world refund behavior.

Contracts and tests

Real-model outputs pass through strict versioned schemas. Invalid, truncated or extra-field responses are rejected, never coerced; every attempt is retained in attempts.jsonl with its status (VALID, INVALID_RESPONSE, INVALID_TOOL_SUBMISSION, PROVIDER_FAILURE, MISSING_CAPABILITY kept distinct). Abstention, invalid response, provider failure, missing capability and deterministic fallback are never conflated. All 42 automated tests pass, including real-provider development contract probes that gate any book execution.

Benchmark (diagnostic, not confirmatory)

Sixteen opaque-ID diagnostic cases compared four approaches on identical source contexts (B and C receive byte-identical inputs; labels stay outside every model context):

Approach Valid result
A — structural only Abstains on semantic families (by design)
B — direct fixed-question review 16/16
C — Pi candidate generation plus μ judgment 14/14 valid (2 invalid decision targets excluded, see below)
D — narrow explicit-template rule Mostly abstains; 2/4 on definition_consistency

Two claim-status C cells were excluded as INVALID_DECISION_TARGET: the typed answer supported a reviewer critique, which the scorer first mapped to support of the manuscript assertion. The raw answers are preserved; decision-target-audit.json records the exclusion, and quality.source_findings version 3 now targets the original manuscript assertion explicitly. The valid-cell figures (16/16 vs 14/14) use different denominators and are not an accuracy claim. Reference labels are agent-authored diagnostics, not independent human gold. No quality improvement is demonstrated. The judgment path measured approximately 2–3× the latency of direct review in this comparison.

The read-only book scans below ran before this appendix was added. Their exact source hashes are retained in the run manifests. Every count in this table comes from results/ch36.jsonl, metric = manuscript_evaluation, with the matching run name.

Run Documents scanned Mechanical issues Comparison candidates Findings with exact source matches
jev-final 35 2 0 2
pi-final 46 0 2 2
pilot-release 2 1 2 3
chapter-final 1 0 0 0

The Pi scan includes a supporting page alongside its numbered chapter files. The Jev issues are duplicate YAML mapping keys in companion metadata. The Pi candidates are exact repeated passages whose usefulness still needs an editorial decision. An exact match confirms the quotation; it does not confirm that repetition is needless. Both book scans recorded unchanged source bytes. Their semantic-model stage is explicitly NOT_RUN.

The protocol is EXPLORATORY. The experiment contract was written before execution, but the author’s instruction prohibited commits. It therefore does not meet the book’s committed-preregistration rule. No promotion of a book-level hypothesis follows from these results.

Coverage (partial by record)

Whole-book semantic evaluation is incomplete and reported as such:

Book Chapter pages completed (best run) Cross-chapter pairs
Jev 7/87 24 sampled (not exhaustive)
Pi 0/83 24 sampled (not exhaustive)

Structural checks ran (Jev 809, Pi 1093); metadata receives deterministic checks only. Completed units mean receipt plus a valid bounded submission — not proof of comprehension or absence of defects. Interrupted runs (capstone02b-20261111, closure-v21) retain their failures; no partial run is presented as complete.

Bring your own provider (bounded)

When you have a running, tool-capable OpenAI-compatible local server, select its endpoint and model explicitly. The exact command and replacement arguments are documented in the package README. --analysis pi-local --provider local uses Pi to prepare candidates and μ to judge bounded questions; --analysis direct-local bypasses Pi for direct review.

The 02B real-model study instead used the configured opencode-go/deepseek-v4.1-flash Pi transport with frozen code/context hashes before dispatch (EXPERIMENT02B.md, experiment02b.mjs). It is not rerun here: the exposed diagnostic cases must not become new held-out tests, and whole-book reruns would need fresh held-out labels plus a new freeze.

Only loopback HTTP is supported. The program refuses redirects and hosted endpoints, imposes request and reading budgets, and records partial failures. It does not discover credentials, install weights or silently select a different provider. Missing model revision and usage remain UNKNOWN. Served revision and billed cost were never observed; catalog estimates are not invoices.

Each family starts in shadow. A proposed promotion rule requires independent author labels, a committed comparison protocol and satisfactory per-family precision, recall, abstention and cost evidence. One successful family does not validate the others. That proposed criterion has not been met here.

Make your editorial decision durable

Open report.md, follow a finding’s evidence spans, and compare its raw judge answer with the policy outcome. Then use the CLI’s review command to record ACCEPT, REJECT, DEFER or INVESTIGATE against the printed finding ID. The package README gives the commands; the acceptance suite demonstrates both acceptance and rejection with synthetic review labels. Those test labels are not your editorial judgments.

The review ledger preserves rejected findings. Each label belongs to a source revision, rubric and task. If you change a chapter, verify marks affected findings stale; a later evaluation creates new evidence. --resume continues an identical run, while --previous can reuse unchanged chapter analysis in a new run. Neither command overwrites the old source record.

What you have established is a useful software boundary: exact evidence, interchangeable bounded providers, explicit uncertainty and an inspectable policy before an editorial action. What you have not established is that a decision model improves manuscript judgment. The evidence supports publishing this as an exploratory case study with a negative benchmark result — and that is a result worth keeping.