← Retrieval From First Principles

Improve and Writer: A Real Retrieval Workflow

Compose the reference retriever with a local consumer while retaining application control over admission, review and side effects.

A manuscript assistant finds a passage relevant to a proposed edit. It then applies the edit because the search score looks convincing. Retrieval has answered a discovery question and the application has mistaken that answer for authorisation to change the book.

The failure is architectural. Better ranking cannot repair a missing approval boundary.

Defending the Retrieval Boundary separated source content from authority. An application must preserve that separation when evidence becomes a model request, a recommendation or a proposed action.

The title names the intended application contexts. Writer and Improve are not integrated. Their current APIs, storage schemas and permissions have not been verified in this batch. The executable example runs entirely inside this repository, using the same contracts that an eventual inspected adapter would need to respect.

Divide responsibilities at the application boundary

The application supplies the information need, a caller-authorised source scope, output requirements, retrieval and admission budgets, and the permitted action set. It also decides whether the request asks for information or proposes a change.

Retrieval returns eligible evidence, qualified references, rank origins, selection details, stop conditions and limitations. It does not acquire permission to edit because it found a related paragraph. Its score does not approve a proposal or certify a model answer.

The application remains responsible for current admission, input assembly, consumer execution, side-effect control, review and approval. Writing and publication require their own authority. Those responsibilities remain even when retrieval and generation are implemented in one process.

Compose the modules we already have

examples/retrieval_application.py provides the ordinary local path:

result = run_request(
    request, tools, consumer, config,
    policy="lexical",
)

The request contains its original question, identity and trusted view, with an optional narrower source allowance. It contains no reference answer or support label. The application calls the existing controller, builds the versioned contract, validates and admits evidence, executes the local consumer and parses its output. Admission uses that same request view and caller allowance. The ordinary call does not obtain a newer external permission state.

Answer grading happens outside that function. The experimental runner loads external answer references only after the ordinary application returns. Gold-evidence conditions are explicitly separate diagnostics, rather than a convenient way to leak labels into the ordinary retriever.

This is the smallest integration needed to observe the final boundary. It adds no server, framework, database, autonomous service or remote model endpoint.

Inspect what crossed the boundary

Every retained call records the original query, retrieval packet, exact selected references, actual admitted units, assembled messages and prompt token IDs. It also retains raw generated text, output token IDs, format status, citations and separate scoring fields.

An application can therefore ask concrete questions. Did the required source enter the selected packet? Did it survive current-view validation? Did it fit the consumer budget? Was the exact text present in the actual prompt? Did the output cite an admitted label? Did the answer match the external reference under the declared rule?

Those records do not answer every question. A prompt trace cannot prove causal evidence use. A valid citation cannot establish entailment. An exact answer match cannot demonstrate human helpfulness. The result preserves these unknowns rather than hiding them behind a single application-success flag.

The real TinyLlama comparison supplies actual consumer outputs for this workflow. Retained output is inspectable evidence of that execution, not a claim that reading the capture generates a fresh answer. Routine replay reconstructs retrieval and tokenised admission while reusing the recorded model output.

Keep failure paths executable

The no-retrieval condition creates an explicit empty evidence input. The constructed missing-procedure need tests what the consumer does when the complete fictional inventory lacks the requested procedure. Related text is not silently treated as the missing instructions.

Revocation and wrong-version tests explicitly supply an updated trusted view and stop the original packet at admission. The contract can refuse newly ineligible references when given that state; the ordinary path does not automatically discover real-world permission changes. A model execution failure has its own error category, while an output that violates the requested format remains a later consumer failure. Neither is disguised as successful retrieval merely because the earlier search ran.

The workflow also records capacity exclusions. If the model receives only one of two useful sources, an evaluator can locate the loss at admission rather than blaming ranking or assuming that selection implied inclusion.

The repository includes an executable inspection of these paths:

python -B examples/run_retrieval_demo.py --case fact --output .artifacts/fact-inspection.json
python -B examples/run_retrieval_demo.py --case missing --output .artifacts/missing-inspection.json
python -B examples/run_retrieval_demo.py --case revoked --output .artifacts/revoked-inspection.json

The first two reconstruct the actual tokenised input and display the retained real model output. The third executes the changed-view admission check and invokes no consumer. Fresh output paths prevent overwriting an earlier observation. These are explicitly replay and validation operations; new generation uses the separate experimental runner.

These paths make the application reviewable. They do not establish unattended recovery, service-level guarantees or correct abstention on arbitrary missing evidence.

What a Writer adapter would need

A future manuscript assistant could request related chapters, source research, historical versions or cited evidence for an editing question. An adapter would first need to establish Writer’s actual interface and identify which source versions and permissions it can expose.

The retrieval result would remain evidence for the editing task. The application would show the source, explain unresolved obligations and distinguish a recommendation from an accepted change. A paragraph relevant to an edit is not necessarily evidence that the edit is correct.

No metadata schema, database table or endpoint is invented here. The local response contract demonstrates the boundary; it is not a tested Writer adapter. Real compatibility remains unmeasured until that interface is inspected and a separately authorised integration is executed.

What an Improve adapter would need

A research-impact workflow could use retrieval to find manuscript passages potentially affected by a new source. It could present the evidence, preserve provenance and make uncertainty visible to a change-planning process.

Discovery would still not establish that a passage needs changing. The source might qualify an earlier claim, conflict with another authority or concern a different setting. A reviewer must inspect the proposed relationship before accepting a change.

This batch creates no proposal in Improve, applies no patch and changes no database. It does not bypass any external review gate. Improve remains not integrated, with its actual interface and approval semantics unverified.

Human control is a state, not a sentence in a prompt

The local application grants the model no action tools and records no writes. It records human approval as not requested or simulated. We do not manufacture an accepted proposal to make an integration demonstration look complete.

An eventual application needs explicit reviewable states: evidence inspected, recommendation considered, change accepted or rejected, authorised action executed. Each transition needs its own record and permission. Reversibility can reduce the cost of an authorised mistake; it cannot supply missing authorisation.

The diagram places the states this demonstration reaches and the states it does not. Nothing after the evidence packet is automatic, and nothing in the local application writes.

    %%{init: {"flowchart": {"wrappingWidth": 470}}}%%
flowchart TD
    S1["Trusted question with a declared scope"] --> S2["Retrieval and admission: evidence packet recorded"]
    S2 --> S3["Consumer produces text; no action tools are granted"]
    S3 --> S4["Trace retained: candidates, packet, admission, output"]
    S4 -.->|"not reached here"| S5["Human review of the recommendation"]
    S5 -.->|"not reached here"| S6["Accept or reject"]
    S6 -.->|"not reached here"| S7["Authorised action executed, with its own record"]
    classDef later stroke:#8b95a1,stroke-dasharray:5 4,fill:#f0f2f5;
    class S5,S6,S7 later;

The local run records human approval as not requested or simulated, which is the honest state rather than a placeholder for approval.

The tested contribution is a local, inspectable route from a trusted question through retrieval and admission to real model output. It shows where a consumer can depend on the evidence contract and where the application retains responsibility. It establishes neither external application compatibility nor complete task effectiveness.

What We Can Now Prove returns to the entire system and asks which of those stronger claims the accumulated evidence actually supports.

A real adapter with a narrower job

The local Pi extension in integrations/pi/ provides a concrete application adapter without claiming Writer or Improve integration. Its three tools call the Python facade for search, exact open and observable explanation. The factory registers capabilities only: no research corpus is injected into a prompt at startup, and no source text can add a new tool. The final chapter loads it explicitly and distinguishes actual loader/tool execution from an unperformed model-provider turn. Writer and Improve remain not integrated.

References and further reading

Background and extensions for this chapter: