Search, Memory and Long Context

Chapter 14 gave the program tools. It can inspect an environment instead of receiving all evidence up front.

Now we face the selection problem:

When the necessary evidence is too large to place directly in the prompt, how should the program decide what to inspect?

Several mechanisms are often collapsed into one word: context. They are different.

1. Put context directly in the prompt
2. Retrieve context before the LM call
3. Let an agent choose retrieval tools
4. Retrieve previous cases / memory
5. Let a model programmatically explore a large context

Each creates a different evidence-selection boundary, with different costs, failure modes, and leakage risks.

1. Context stuffing

The simplest repository repair strategy is:

issue
+
entire repository dump
      ↓
LM

This can work for tiny repositories, and long-context models can push that boundary surprisingly far. The problem is not a universal context length at which stuffing suddenly fails. The problem is that cost, latency, attention competition, truncation risk, and provenance burden all grow with the constructed context.

A stuffed-context run is reproducible only if the context is treated as an artifact rather than an informal preprocessing step:

2. Retrieval before the LM call

A more controlled shape is:

issue
   ↓
retriever
   ↓
top-k evidence
   ↓
program

The retriever may be lexical, embedding-based, graph-based, symbol-aware, reranked, or hybrid. Retrieval is not just “top-k documents”; it is a versioned decision procedure that determines which evidence reaches the program.

Its run record should capture enough state to reproduce that decision:

CoCoder’s repository-analysis and candidate-evaluation systems reflect this kind of concern. A candidate patch case records repository revision, input references, validation contract, and fingerprints.

That revision boundary is essential. A retrieval result from commit B cannot silently become evidence for a decision that claims to have been made against commit A. If indexes are built asynchronously, the index snapshot must also be traceable to the source revision it represents.

The goal is not to preserve every incidental implementation detail. It is to preserve enough lineage to answer: what evidence could this program actually have seen when it made this decision?

3. Memory and CBR have different time semantics

Memory retrieval is still retrieval. What changes is the evidence source.

Retrieval from the current repository asks:

4. Agent-directed retrieval

Fixed retrieval usually commits to an evidence-selection procedure before the task model sees the retrieved context. Agent-directed retrieval makes evidence selection adaptive:

5. RLM-style exploration

Current DSPy marks dspy.RLM as experimental. Its default implementation keeps large inputs as variables in a Python interpreter and gives the LM metadata about those variables rather than placing the entire values in every LM prompt. The model iteratively writes Python to inspect the data and can call llm_query(...) or llm_query_batched(...) for focused semantic sub-analysis.

The current default PythonInterpreter uses Deno + Pyodide in a local WASM sandbox. RLM also accepts custom tools, a separate sub_lm, and a configurable interpreter factory; network access can be enabled through interpreter configuration. “Sandboxed” therefore describes a capability boundary that must itself be configured and audited, not a guarantee that arbitrary custom tools or network-enabled interpreters are harmless.

Current RLM also exposes explicit execution limits such as max_iters, max_llm_calls, and max_output_chars. Those limits, the main LM, sub_lm, tool set, interpreter configuration, and input fingerprints all belong in the run record.

The stable idea is broader than the current API:

large context remains external
        ↓
model sees available variables/tools
        ↓
model writes code to inspect pieces
        ↓
focused evidence accumulates
        ↓
final structured answer

These mechanisms are not mutually exclusive, but they place adaptation in different parts of the system:

An agent can call a RAG retriever. An RLM can receive custom tools. A retrieval system can use an LM-generated query. The useful distinction is therefore where evidence-selection decisions are made and what must be recorded to reproduce them, not which marketing label the system uses.

RLM is not the foundation of this book because its API is explicitly experimental. The durable lesson is independent of that class name: keep large state outside the immediate token stream when useful, expose bounded inspection operations, and record the path by which evidence entered the decision.

6. Repository repair as context problem

A repair program may need:

pyproject.toml
package structure
symbol definitions
callers
tests
configuration
git diff
previous similar repairs

It almost never needs every file. The program should produce an evidence packet:

Later evaluation should ask two separate questions:

What Usually Goes Wrong

Conclusion

We gained a vocabulary and an audit model for large evidence. Direct context, fixed retrieval, historical memory, agent-directed retrieval, and RLM-style exploration place evidence selection in different parts of the system and therefore require different provenance.

We removed the assumption that a larger context window is a substitute for evidence selection—or that retrieval is reproducible merely because the final documents were saved.

The program can now search, remember, and programmatically explore large environments. That creates the next danger: adaptive evidence selection may discover information that the experiment was supposed to keep hidden.