Chapter 04 of 17

From Retrieval to Persistent Understanding

Concepts

Chapter 4 β€” From Retrieval to Persistent Understanding

Source: 04-chapter.md

What this chapter is really about

Underneath the GraphRAG machinery, this chapter is about the moment interpretation becomes state. Its deepest content is the transient/persistent distinction: strong RAG already understands at query time, so the question is never whether the system comprehends but whether any comprehension survives the query. Technically it is about building a derived graph and measuring whether it pays; philosophically it is the book’s first demonstration that storing intelligence and storing mistakes are the same act. The chapter asks: what does persistent understanding buy once excellent transient understanding already exists?

Current thesis

Explicit claims

  • Strong RAG already interprets retrieved evidence at query time; the RAG-understands-nothing framing is a strawman this chapter refuses.
  • RAG interprets history for this query; persistent graph memory preserves some interpretation for future queries.
  • Derived memory is a hypothesis about history, not a rewrite of it; raw sources stay canonical and the graph stays rebuildable.
  • On the current local fixture, graph conditions match but do not beat strong RAG on decision accuracy (adjudicated 8–9/9 across full-coverage conditions), at 2.2–2.6Γ— context and roughly 7–25Γ— latency plus a 999 s index build.
  • Persistent errors are stored errors: the frozen index splits J. LINDQVIST/J. LINQVIST and EVENT-STORE/EVENT STORE, with the typo node owning three relationships.
  • Source mapping (derivation provenance) is complete at zero orphans and is explicitly not evidential support; support belongs to Chapter 7.
  • Verdict: Type C with a Type B core β€” developmental, pending repaired scorers, relational/synthesis families, and consistency measurement.

Implied claims

  • Some interpretations are stable enough across queries to earn storage; the chapter bets entity/relationship/claim/community structure is among them without proving the boundary.
  • The one relational bright spot generalises to a family where structure composes across artifacts; unmeasured, stated as obligation.
  • Community-report synthesis will matter at scales where no passage is the answer; untested on the current fixture by missing Global cells.

Not yet established

  • Whether adjudicated parity survives repaired mechanical scoring at matched raw budgets (the budget-composition confound is documented, not removed).
  • Cross-query consistency: whether stored distinctions answer related questions more compatibly than independent reconstructions.
  • Global-synthesis quality in any direction; Global mode ran none of its home-territory tasks.
  • DRIFT quality or cost beyond one 90-second timeout.
  • Index rebuild stability under identical configuration (extraction drift unmeasured).

What the chapter already gives us

  • The transient/persistent distinction. The book’s most reusable conceptual correction: memory begins when interpretation becomes state, not when comprehension first occurs.
  • The three layers. Raw source, persistent derived interpretation, query-relative use β€” with the rule that Layer 3 must never be stored as permanent truth.
  • The derivation/support boundary. Source mapping answers where a derived object came from; Chapter 7 answers whether evidence licenses a claim. Stated early enough to prevent a whole class of later confusion.
  • The typo node. J. LINQVIST as the concrete exhibit of persistent error β€” more instructive than any taxonomy of extraction failure.
  • The honest cost table. Latency, context, and build cost reported from artifacts alongside quality, refusing the score-only comparison.
  • The proposal/preference/decision migration. The old chapter’s best example reframed from representation-failure proof to reuse investigation, preserving continuity without preserving the overtaken conclusion.

Where the current treatment stops

  • Consistency is specified but unmeasured; the six-question reuse demonstration is prose, not runs.
  • The synthesis family it calls for does not exist; Global’s value proposition is argued, not tested.
  • Scorer aliasing and citation normalisation defects are documented workarounds inside the analysis, not repaired instrument code.
  • The update model is full rebuild; incremental economics are contrasted (LightRAG) but not implemented or measured.
  • Entity-resolution repair is diagnosed (health flags the splits) but no resolution mechanism is attempted.

The deeper territory

  • Amortisation accounting. Indexing at 999 s for 20 documents only pays over query volumes the fixture never approaches. The general question β€” under what query distribution does persistent derivation amortise? β€” recurs for consolidation (Ch 15) and compression (Ch 16).
  • Error persistence as a first-class failure class. Transient misreading versus stored misrepresentation need different detection (health/provenance) and different repair (rebuild versus re-prompt). The book’s failure taxonomy gains a persistence axis.
  • The storage/computation frontier. LazyGraphRAG (derive per query, store nothing) versus full GraphRAG (derive once, store everything) is the sharpest form of this chapter’s question, unimplemented here and reserved as the null hypothesis.
  • Version identity of understanding. Each derived build is a versioned hypothesis; downstream consumers (association, routing, lineage) pin versions implicitly. What depending-on-a-version means for invalidation is Chapter 7–8 territory glimpsed early.

Concepts worth developing

Transient versus persistent understanding

Idea. Comprehension that exists only inside one inference versus comprehension stored as state that influences later queries. The memory threshold is persistence, not cleverness.

Why it matters. It dissolves the retrieval-versus-understanding debate that otherwise misdirects every structured-memory proposal into beating a strawman.

Connection to the current chapter. The opening correction and the closing principle; everything between is the measurement of whether persistence pays.

Broader implication. Later layers (association, routing, lineage, consolidation) are all persistent-state proposals and must each pass the same audit: what is stored, what it costs, what its stored errors are.

What remains unresolved. The boundary principle for what deserves storage versus per-query computation.

Derived memory as versioned hypothesis

Idea. Every derived build carries corpus hash, models, prompts, chunking, and commit; changing any of them creates a new version rather than mutating understanding.

Why it matters. Without version identity, stored errors cannot be attributed, rolled back, or invalidated downstream.

Connection to the current chapter. Manifest discipline, rebuild rule, fallback architecture.

Broader implication. Chapters 7–8 need versioned derivation to localise errors and resolve supersession over derived claims.

What remains unresolved. Incremental update identity: what a partial rebuild means for version continuity.

Derivation provenance versus evidential support

Idea. Where a derived object came from (source mapping) is distinct from whether its sources license a claim (support). Connectivity implies neither truth nor justification.

Why it matters. Conflating them lets well-travelled routes masquerade as reasons β€” the error Chapters 5 and 7 are built to refuse.

Connection to the current chapter. Layer 2 boundary; zero-orphan provenance result explicitly fenced.

Broader implication. The support-group semantics of Chapter 7 and the validity resolution of Chapter 8 both presuppose this separation.

What remains unresolved. How much derivation detail lineage must retain to localise errors (Chapter 7’s open provenance-retention question).

Important distinctions

  • Transient interpretation versus persistent interpretation.
  • Raw evidence versus derived state versus query-relative use.
  • Derivation provenance versus evidential support.
  • Graph edge versus support edge.
  • Source-aware interpretation versus source-guaranteed truth.
  • Indexing amortisation versus query-time derivation (the LazyGraphRAG null).
  • Mechanical score versus adjudicated quality (and why both are reported).
  • Budget-matched totals versus budget-matched composition.
  • Enabling infrastructure (Type C) versus direct QA victory (Type A).
  • Structural health versus semantic correctness.

What mechanism would make this work?

Matched final-reader comparison: same corpus (ch3-fixture-v0.1), same reader (llama3.1:8b), same total context budget; graph conditions add one GraphRAG call with derived synthesis capped at one third of context. Mechanical scorers over admitted raw evidence; adjudicated rescoring under a stated rescue rule committed as analysis. Error entry: alias-list gaps, citation-format normalisation, budget-composition effects, Global coverage gaps, unmeasured DRIFT/consistency/synthesis. Missing: repaired scorers, relational and synthesis families, consistency metric, rebuild-stability probe.

Connections to the rest of the book

  • Chapter 3 supplies the reader, the corpus discipline, and the rival that must be beaten fairly; Chapter 4 returns the verdict that it was not beaten.
  • Chapter 5 consumes the snapshot through the graph adapter and traverses what Chapter 4 stored, typo nodes included.
  • Chapter 6 routes among the modes Chapter 4 measured, with the cost table as its price list.
  • Chapter 7 verifies the derivations Chapter 4 stored and decides which ones license claims.
  • Chapter 8 inherits versioned derived state as one more thing whose current truth must be maintained.
  • Chapters 15–18 replay the amortisation and reinforcement questions for consolidated and adaptive state.

Beyond the current book

  • Database view maintenance: materialised views as persistent derivation with invalidation β€” the closest mature engineering analogue of the derived graph and its rebuild economics.
  • Provenance semirings in databases: where-provenance versus why-provenance, the formal ancestor of the derivation/support split.
  • Speech-act theory and argument mining: the proposal/preference/decision distinctions the chapter reuses rather than re-derives.

No new verified paper beyond the chapter’s own references. The planning research matrix (Microsoft GraphRAG paper, LazyGraphRAG, DRIFT, LightRAG, HippoRAG 2, RAPTOR, Claimify, VeriTrail, BenchmarkQED, two surveys) is recorded in research/bibliography.yaml with peer-reviewed/preprint/vendor-reported/documentation statuses; vendor-reported figures are labelled as such in prose.

Possible future claims

Already supportable

  • The frozen index contains the reported counts with complete source mapping and named stored errors (artifact-backed).
  • Graph query modes cost multiples of baseline latency and context at parity decision quality on this fixture (artifact-backed with stated latency-accounting limits).

Plausible but needs development

  • Relational composition is where stored structure helps (one case; needs a family).
  • Synthesis at scale favours community reports (argued; needs the synthesis family).
  • Cross-query consistency favours persistent distinctions (specified; unmeasured).

Speculative

  • The amortisation crossover: query volumes at which indexing pays (no cost model yet).
  • Lazy derivation as sufficient substitute for persistence on this workload (reserved null hypothesis).

Claims worth attacking

  • “Basic matches the baseline.” Counter: composition confound β€” Basic surrenders a third of raw budget, so parity under handicap could hide either superiority or fragility. Matched-composition rerun decides.
  • “Adjudicated 9/9 for Basic.” Counter: post-hoc rescue rule, however stated, is not a frozen scorer. Repaired-scorer rerun decides.
  • “Zero orphans means provenance works.” Counter: mapping completeness is not mapping correctness; a derived object can map to the wrong artifact cleanly. Correctness audit decides.

Tensions and counterarguments

  • Persistence versus freshness: everything stored is stale the moment history moves; full-rebuild economics may dominate the quality story at real update rates.
  • Structure versus reader strength: a stronger reader may recover compositionally what the graph stores, shrinking Type C toward Type B.
  • The LazyGraphRAG challenge: if per-query derivation matches persistent quality at lower total cost, storage earns nothing here.
  • Single-extraction-model generality: all findings are conditional on ministral-3:8b extraction quality, with the mistral-small timing abandonment already showing model sensitivity.

Examples and thought experiments

  • The seventh question: six related questions answered compatibly from stored structure, then one question where the stored distinction is wrong β€” does the system repeat the stored error six times consistently?
  • The rebuild lottery: same corpus, same frozen config, rebuilt index β€” how many typo nodes survive, how many new ones appear?
  • The update arrival: a reversal lands after indexing; which answers change, which stay wrong until rebuild, and does anything mark them stale?

Potential demonstrations or experiments

E4-A strong RAG vs Basic at matched composition; E4-B Local on entity/relationship tasks; E4-C small synthesis family; E4-D cross-query consistency over the six-question reuse set; E4-E provenance completeness audit (done, zero orphans); E4-F injected false-relationship downstream observation; E4-G cost profile (done for build/query latency/context); E4-H rebuild stability on a controlled subset. Specified in the experiment slot; A–D unrun.

Research questions this chapter creates

  • What deserves storage versus per-query derivation, stated as a decision rule rather than an intuition?
  • At what query volume and update rate does full-index derivation amortise against lazy derivation?
  • How is cross-query consistency measured without smuggling cross-mode agreement into correctness?
  • What is the rebuild-stability distribution of LLM extraction under frozen configuration?

Architectural implications

  • The backend protocol is the chapter’s durable artifact; packages behind it are replaceable.
  • Fallback to raw RAG is structural, not a demo path β€” required by the Chapter 6 harm finding.
  • Versioned derived state becomes an assumption of Chapters 5–8; invalidation policy is the open interface.
  • Entity resolution is now a persistent-memory reliability problem, not a retrieval-tuning problem.

How would we know this works?

Type A needs material relational/global/consistency gains at acceptable cost on repaired scorers; Type B needs parity-or-better for strong RAG with graph costs documented (this run’s shape); Type C needs downstream consumption demonstrated (Chapters 5–7 consume the graph); Type D needs derived harm outweighing benefits. Current evidence: B on direct QA, C on architecture, A/D unsupported.

The chapter at its highest level

The ideal version teaches persistence as the memory threshold, demonstrates it with a real versioned index, measures it against a rival built to win, reports the rival’s survival as a finding rather than a failure, exhibits the stored errors honestly, and hands later chapters both a substrate and a cost ledger. The current version does all of these at fixture scale with explicitly labelled developmental status on quality claims.

Discussion

Start here

  • If a stronger reader recovered relational composition from raw passages, would anything in the graph still earn storage on this workload?
  • The typo node owns three relationships. Which downstream consumer β€” association, routing, lineage β€” is most harmed, and which detects it first?
  • Where exactly should the storage boundary sit: which interpretations are stable enough to persist, and who decides?

Push the idea further

  • Does the amortisation crossover exist at real update rates, or does full rebuild make persistence a luxury good?
  • If lazy derivation matched persistent quality, would the graph retain any role beyond the downstream consumers already built on it?
  • Can consistency be measured without rewarding systems for repeating their own stored errors compatibly?

Decisions we need to make

  • Whether to fund the repaired-scorer matched-composition rerun before Chapter 8, and what verdict change would alter the build plan.
  • Whether entity resolution enters as its own mechanism or stays a rebuild-quality obligation.
  • Whether DRIFT gets any fixture measurement or stays pending until a workload needs it.

Claims worth attacking

  • “Persistence is the memory threshold.” Counter: versioned raw history with excellent per-query derivation might satisfy every behavioural test without stored interpretation β€” the LazyGraphRAG position.
  • “Adjudication rescues truth from scorer fragility.” Counter: post-hoc judgement is the instrument’s evaluator-privilege failure wearing a lab coat; only a frozen repaired scorer counts.
  • “Type C justifies the index.” Counter: downstream use of a structure is not evidence the structure is good β€” Chapters 5–7 could be consuming an expensive habit.

New ideas worth exploring

  • Amortisation accounting as a standing chapter obligation: every stored-state proposal ships with its crossover estimate.
  • The persistence axis of the failure taxonomy: transient versus stored error as a first-class diagnostic split.
  • Lazy derivation as the permanent null hypothesis for every persistence claim.

Explain this chapter with AI

Copy this prompt into ChatGPT, Claude, Gemini, a local model, or another AI.

Apply this chapter with AI

Copy this prompt into ChatGPT, Claude, Gemini, a local model, or another AI.

Chapter 3 built the strongest conventional retrieval the book could assemble and left it standing. That result constrains this chapter before it begins. Any claim of the form retrieval finds text but does not understand it would be false about the system Chapter 3 actually built: a hybrid retriever, a cross-encoder reranker, and a capable reader already perform substantial interpretation at query time. The reader distinguishes proposals from decisions when the passages allow it, follows rationale across artifacts, and abstains when evidence runs out. That is understanding, deployed per query and discarded afterwards.

The question is therefore sharper than the usual argument for structured memory:

What do we gain by preserving AI-derived understanding as reusable state rather than reconstructing it from raw evidence independently on every query?

The chapter’s answer has two halves. The first is architectural: a persistent derived graph built with Microsoft GraphRAG, inspectable, versioned, and rebuildable, with raw history remaining authoritative underneath it. The second is experimental: on the current local workload, strong conventional RAG remains extremely competitive, the graph does not produce a general quality win, and its costs are large. Its value is conditional β€” relational reuse, corpus-wide synthesis, and downstream mechanisms that consume persistent structure β€” and its mistakes, unlike a reader’s, persist.

RAG already understands

The strawman version of this chapter would say that retrieval matches words while graphs understand meaning. Nothing in the book’s own evidence supports that division. Chapter 3’s reader receives up to six passages and reasons across them jointly; the reranker scores query and passage together before the reader ever sees them. When the frozen comparison in this chapter asks what event store should new services use?, the baseline answers PostgreSQL, cites the decision record, and does so from the same raw passages any graph condition sees. Interpretation happened. It happened at query time, inside one inference, and it left no trace.

That observation reframes the entire enterprise. The transition this book studies is not from dumb matching to smart comprehension. It is from transient interpretation to persistent interpretation:

RAG interprets history for this query. Persistent graph memory preserves some of that interpretation for future queries.

Everything the graph stores β€” an entity, a relationship, a claim, a community summary β€” is an assertion of the form an AI system read these passages and concluded this. Each one amortises repeated reading: the tenth question about the event-store migration need not re-derive that PostgreSQL replaced SQLite from raw sessions if a relationship already says so. Each one also fossilises fallibility: if the extraction was wrong, the tenth question inherits the error without re-reading the evidence that would have corrected it. Reusable intelligence and reusable mistakes are the same mechanism. The experiment measures both.

What disappears after the answer?

Consider what the baseline knows at two moments. During inference on why was PostgreSQL chosen?, its context contains the contention report, the benchmark, the incident, and the decision record, and its activations transiently encode the connection between them. One second later the answer is delivered and that connection is gone. The passages remain in the index. The understanding does not.

Three consequences follow. First, every query pays the full price of interpretation: retrieval, reranking, and reading start from raw text regardless of how many times the same connection was already derived. Second, nothing accumulates: twenty questions about the migration leave the system exactly as knowledgeable, in any stored sense, as zero questions did. Third, consistency is accidental: whether what did we decide?, why?, who proposed the alternative?, and what was rejected? receive mutually compatible answers depends on the reader reconstructing the same distinctions four times independently.

A persistent derived memory is the hypothesis that storing some interpretation is worth its cost. Note what the hypothesis does not say. It does not say retrieval is weak β€” Chapter 3 forbids that reading. It does not say every interpretation should be stored β€” query-relative judgements belong to the query, as the three-layers section shows. It says that some interpretations are stable enough across queries to earn storage, and that the book should measure whether that storage pays.

Make understanding persistent

The architecture under test keeps two stores with different epistemic status:

    flowchart TD
    SRC[original sources] --> RAG[conventional RAG]
    SRC --> IDX[GraphRAG indexing]
    IDX --> DER[derived graph]
    RAG --> READER[reader]
    DER --> READER
    READER --> ANS[answer]
  

Raw sources are canonical. The derived graph is a hypothesis about history, not a rewrite of it. If the graph is wrong, the repair is to fix or rebuild the derived state β€” never to edit the source, which remains the ground the system falls back to and the auditor inspects. That invariant, lossless history underneath and constructive memory above it, governs every later chapter: association traverses the graph but cites the sources, routing chooses graph modes but keeps raw retrieval available, evidence lineage treats derivation as something to verify rather than trust.

Derived memory is a hypothesis about history, not a rewrite of history.

The comparison this chapter runs is therefore:

source history
    ↓
retrieval β†’ context β†’ reader interprets evidence for this query β†’ answer

against:

source history
    ↓
AI interpretation during indexing
    ↓
persistent derived structure (entities, relationships, claims, communities)
    ↓
query over raw + derived state β†’ answer

Same corpus, same reader model, same final context budget. The only difference is whether a prior act of machine interpretation survives between queries.

Raw evidence and derived state

Three layers need distinct names because they fail differently:

Layer 1 β€” raw source. What actually existed: a session transcript, a commit entry, a decision record, a benchmark note. Layer 1 is authoritative. It can be incomplete or misleading as a record, but it is not the system’s opinion about anything.

Layer 2 β€” persistent derived interpretation. What the indexing system inferred: this span mentions PostgreSQL; PostgreSQL replaced SQLite here; these entities belong to one community; this claim is supported by those units. Layer 2 is reusable and versioned. It is also where extraction errors live permanently until rebuilt.

Layer 3 β€” query-relative use. What matters for the current question. The same artifact is decision evidence for what did we decide?, historical background for what runs in production now?, and irrelevant for the caching question. Layer 3 is computed per query and must not be stored as permanent truth, or one question’s framing becomes every later question’s prejudice.

A deliberate terminological boundary follows, and Chapter 7 will enforce it. Layer 2 records derivation provenance: this graph object came from those text units in those artifacts. It does not record evidential support: whether those sources actually license the eventual claim. A relationship row reading PostgreSQL β†’ SQLite is a stored assertion about what the extractor inferred, not a support edge. Graph connectivity must never be read as truth, and a source mapping must never be cited as justification. Chapter 4 answers where did this derived representation come from?; Chapter 7 answers why does that evidence license this claim?

Why a graph?

Persistent interpretation could take many forms: summaries, timelines, tables, embeddings with metadata. The graph form earns its place through three properties, stated here as established background from the survey literature (Peng et al., ACM TOIS 2025; Zhang et al., preprint 2025), not as findings of this book.

First, project history is relational. Decisions supersede proposals, incidents motivate migrations, benchmarks replicate one another, runbooks echo decisions. A flat list of passages stores these connections nowhere; a graph stores them as first-class objects with descriptions and weights.

Second, relations compose across artifacts. No single passage states the contention observed in session-014, confirmed by the benchmark in session-019, caused the importer failure in incident-021, which motivated the decision in adr-007. That chain spans four artifacts. Relationships plus communities give the system somewhere to put multi-hop structure that no chunk contains whole.

Third, communities summarise at scale. When the corpus grows beyond any context window, per-passage retrieval degrades into sampling; community reports offer precomputed corpus-wide summaries organised by topic rather than by file. Whether that helps on real questions is exactly what the Global condition tests.

The estimator of these benefits must be stated alongside the mechanism, because the book’s later verdict depends on it: every benefit above is purchased with LLM inference at index time, stored state to maintain, and a new class of persistent error.

Build it with GraphRAG

The implementation is Microsoft GraphRAG, pinned at version 3.1.2, behind a backend abstraction the book owns. The abstraction matters more than the package: memory is defined here as persistent AI-derived structured understanding, and GraphRAG is the experimental implementation. A later system β€” incremental, lighter, hand-built β€” replaces the backend, not the book’s definition.

class StructuredMemoryBackend(Protocol):
    def index(self, force: bool = False) -> IndexReport: ...
    def query_basic(self, question: str) -> GraphQueryResult: ...
    def query_local(self, question: str) -> GraphQueryResult: ...
    def query_global(self, question: str) -> GraphQueryResult: ...
    def query_drift(self, question: str) -> GraphQueryResult: ...
    def query(self, question: str, mode: str) -> GraphQueryResult: ...
    def snapshot(self) -> GraphSnapshot: ...
    def manifest(self) -> dict: ...

Book result. The StructuredMemoryBackend protocol in solution/graph_memory/graphrag_backend/backend.py exists with Microsoft GraphRAG 3.1.2 as its first implementation. All callers β€” the experiment adapter, the CLI, the health checks, the Chapter 5 adapter β€” program against the protocol.

What the package contributes, verified against the installed version rather than marketing text: documents, text units, entities, relationships, optional claim covariates, Leiden-hierarchy communities, community reports, and embeddings, persisted as a parquet index. Four query modes ship in the package: Basic (vector search over text units, the closest internal analogue of conventional retrieval), Local (entity-centred search combining graph neighbourhoods, text units, and community context), Global (map/reduce synthesis over community reports), and DRIFT (a primer over community reports followed by iterative local follow-ups). The chapter describes only modes actually exercised; DRIFT’s experimental status is reported honestly below.

Source handling preserves context the interpretation depends on. A statement such as PostgreSQL is the right choice means something different in a chat proposal, an assistant reply, a decision record, a benchmark note, a commit message, and a database export. The source adapters in solution/graph_memory/sources/ normalise every input into a shared envelope β€” artifact identifier, source type, path, timestamp, actors where stated, content, hash, metadata β€” reusing Chapter 3’s source identity rather than inventing competing identifiers. Source type affects what kind of historical object the system possesses; it does not make the content correct. That principle is load-bearing: nothing in the pipeline treats decision records as true by virtue of being decision records.

Provenance mapping walks every derived object back toward canonical artifacts through text units and input documents. Anything unreachable is recorded as an orphan with its broken link named, usable as a hypothesis but never as evidence. Health checks report counts, mapping coverage, near-duplicate entities, indexing failures, and build freshness β€” structural health, explicitly not semantic correctness. The reader inspects all of it without a model server:

python -m graph_memory.cli index
python -m graph_memory.cli health
python -m graph_memory.cli inspect --counts
python -m graph_memory.cli inspect --entity PostgreSQL
python -m graph_memory.cli query "..." --mode basic
python -m graph_memory.cli query "..." --mode local
python -m graph_memory.cli query "..." --mode global

Look inside the derived memory

Book result. The frozen index over the Chapter 3 fixture corpus (ch3-fixture-v0.1, corpus hash 04aac354) contains 20 documents, 20 text units, 39 entities, 65 relationships, 56 claims, 6 communities, and 5 community reports. Indexing ran 999 seconds of wall time, 55 LLM responses, and roughly 128,000 tokens, using ministral-3:8b for extraction and bge-m3 for embeddings. One community (community 2) has no report: the summariser’s output failed schema validation, recorded in the index report as an extraction failure. Build counts and provenance below are read directly from the committed index, not estimated.

The content is recognisably the project’s history, reorganised. SQLITE (degree 17) and POSTGRESQL (degree 8) anchor the migration; ADR-007, SESSION-044, SESSION-040, and the benchmark entities carry the decision and its evidence. A first community report summarises the event-store migration from SQLite to PostgreSQL through contention and benchmarking; another summarises the Redis rejection. So far, so much like a competent reading of the corpus β€” which is precisely the point. The graph stores what a good reader would conclude, so that later queries need not conclude it again.

Then the mess, which the chapter shows rather than tidies. Entity resolution failed in at least two places the health check flags. J. LINDQVIST (degree 4) and J. LINQVIST (degree 3) are the same person, split by a one-letter extraction typo β€” and the typo entity owns three relationships, including J. LINQVIST β†’ REDIS for the Session-040 caching proposal. EVENT-STORE and EVENT STORE are the same system under two spellings, stored as two nodes of different declared kinds. A. NOVAK and A.NOVAK repeat the spacing-split pattern. These are not query-time mistakes that vanish on the next question. They are stored, versioned, queryable mistakes, and every downstream consumer β€” association in Chapter 5, routing in Chapter 6, lineage in Chapter 7 β€” inherits them until the index is rebuilt. The chapter’s central failure exhibit is not a wrong answer. It is a wrong node.

Provenance, by contrast, is complete:

Book result. All 166 derived objects (39 entities, 65 relationships, 56 claims, 6 communities) map back to canonical source artifacts. The orphan rate is zero in every kind. Source mapping tells us where each derived object came from; per the Layer 2 boundary, that is derivation provenance, not evidential support.

Health verdict: structurally unhealthy for exactly one reason β€” the missing community-2 report. The sole indexing failure is named, logged, and preserved rather than repaired silently, because a rebuilt-silent index would break the versioning invariant the next section states.

One source, several questions

The old version of this chapter used the proposal/preference/decision history to argue that flat retrieval discards role and therefore an event schema is required. The stronger baseline overturned the therefore: Chapter 3’s reader distinguishes discussion from decision directly from well-retrieved passages on most of the fixture. The example is retained with its conclusion changed. It now investigates reuse rather than failure.

The canonical history runs Monday to Thursday: session-031 proposes moving the event store to PostgreSQL, session-033 prefers SQLite, adr-007 decides PostgreSQL. Six related questions address the same history:

What did we decide?
Why?
Who proposed the alternative?
What was rejected?
What evidence supported the decision?
What systems are connected to it?

Strong RAG answers each by reconstructing the distinctions fresh: retrieving the three passages, reading headers and status lines, and assigning roles per query. The persistent alternative answers from stored structure: entities for the people and systems, relationships for proposal and supersession, claims for the contention evidence, the community summary for the rationale. The empirical question is whether stored distinctions are more consistent across the six questions than six independent reconstructions β€” and what happens on the seventh question, where the stored distinctions are wrong.

Cross-query consistency of this form is unmeasured in the current runs; the runner records per-case answers but infers nothing across them, and the chapter labels consistency an explicit pending obligation rather than smuggling agreement across query modes into a metric. The claim it does test is narrower: whether graph-backed conditions answer the individual questions as well as strong RAG, at what cost, and with what new errors.

Basic Search is vector search over the index’s text units followed by the same reader that serves the baseline β€” the closest internal comparison to conventional retrieval, differing mainly in chunking and in what text the index holds. Expect it to behave most like Chapter 3, and treat any divergence as a fact about the index pipeline rather than about graphs as such.

Local Search centres on entities: resolve the question’s entities, gather their neighbourhoods, and combine graph context with text units and community summaries. This is the mode whose design best matches decision and relational questions β€” what did we decide about X?, which incidents influenced the decision? β€” because those questions name entities whose stored neighbourhoods should contain the answer’s parts.

Global Search never retrieves passages. It maps over community reports and reduces the partial syntheses into an answer. Its natural territory is corpus-wide synthesis β€” what major architectural themes emerged? β€” where no single passage is the answer and the work is compression across topics. Judged on exact local lookup (what did adr-007 decide?), it is the wrong tool by construction, and the chapter refuses to score it there as though that were informative. The fair test routes global questions to Global Search and local questions to local modes, then reports each mode on its own territory alongside the baseline everywhere.

DRIFT

DRIFT Search combines a community-report primer with iterative local follow-ups. Its status in this chapter is: unmeasured on the fixture suite. A single diagnostic attempt in the audit runs timed out at the 90-second query budget, and the frozen comparison contains no DRIFT cells. Chapter 6 independently records DRIFT as unmeasured in its routing matrix. The chapter does not estimate DRIFT quality from documentation, does not run a full DRIFT matrix for completeness at prohibitive per-query cost, and leaves DRIFT explicitly pending with the timeout artifact committed. Time and cost are experimental variables, and an unmeasured mode is reported as unmeasured.

The experiment

Book hypothesis (pre-registered). Preserving interpretation as a persistent derived graph will match strong RAG on local decision questions, improve relational and corpus-wide synthesis questions where structure composes across artifacts, keep per-answer provenance mappable to raw sources, and cost substantially more in indexing and query latency. Four verdict types were registered before analysis: (A) persistent structure earns a core role through material relational/global/consistency gains at acceptable cost; (B) strong RAG remains sufficient and graph memory stays optional; (C) structure enables later mechanisms without improving direct QA; (D) persistent derived state is net harmful and graph memory is demoted to experimental. The verdict may combine categories.

The frozen comparison (experiments/benchmark/runs/ch4-20260919T205622Z) runs 14 tasks from ch4-tasks-v0.1 over the Chapter 3 fixture corpus, across seven families β€” locate, decision, provenance, temporal, use, relational, global β€” with the llama3.1:8b reader shared by all conditions. The graph conditions (Basic, Local, Global) add one GraphRAG query per task and admit the derived synthesis into at most one third of the final context, with raw passages filling the remainder; source-recall scoring covers only directly admitted raw evidence, so derived lineage can never inflate a recall score. Global ran 8 of the 14 tasks; its six missing cells include both global-family and both relational-family tasks β€” its home territory untested, a coverage defect the analysis does not forgive.

Two fairness qualifications must be stated before any number is read. First, total final context is matched but its composition is not: graph conditions surrender a third of raw budget to the derived block, so their admitted-source recall is structurally disadvantaged relative to the baseline by construction. A recall gap between baseline and graph conditions partly measures that budget split, not retrieval quality. Second, the unsupported-source scorer flags chunk-identifier citations (e.g. incident-021.md:65392e92…) as fabricated while accepting plain source identifiers, which penalises graph conditions differentially for a citation-format difference rather than a grounding difference. Unsupported-source rates are therefore reported but not compared across conditions. Both defects are recorded as instrument obligations: alias lists and citation normalisation need repair before any rerun counts as publication-grade.

What happened

Mechanical scores first, exactly as the frozen artifacts record them (decision exactness over the 9 applicable cases; source recall over all tasks; context tokens estimated; latencies measured wall time):

condition tasks source recall decision exactness mean context tokens mean latency
Chapter 3 best 14 1.000 0.889 (8/9) 580 ~7 s reader-sideΒΉ
Graph Basic 14 0.819 0.889 (8/9) 1317 51.5 s
Graph Local 14 0.819 0.667 (6/9) 1250 92.7 s
Graph Global 8 0.812 0.800 (4/5) 1525 196.1 s (max 585 s)

ΒΉ The baseline cases record component latencies (retrieval plus roughly 5 s generation) rather than end-to-end totals; graph conditions record end-to-end totals including the GraphRAG call. Latency ratios are therefore approximate and, if anything, flatter the graph conditions’ overhead.

Developmental observation. The mechanical decision gaps are dominated by scorer normalisation, not answer quality. Three cells decide the table’s shape: the why-question fixture carries no alias list, so paraphrases of concurrent-write contention (concurrent writes, high write loads) fail while the hyphenated original passes β€” failing all three graph conditions on answers that state the causal mechanism correctly; the Local Redis answer (rejected the proposal to introduce Redis) misses an alias (rejected the redis proposal) by word order on a substantively correct answer; the baseline’s relational answer names two genuine ledger influences by identifier without stating any mechanism and fails the same way.

Adjudicated rescoring under one stated rule β€” rescue only cells whose answers contain the expected mechanism in paraphrase the alias list omits, or near-verbatim word-order variants; no rescue for vague or mechanism-free answers β€” gives best 8/9, Basic 9/9, Local 8/9, Global 4/5 on its covered tasks. The adjudication is committed alongside the frozen run as analysis, not as a modification of frozen data. The largest gap between any two full-coverage conditions is one case in nine. That is noise, not victory, in any direction.

Per-family detail sharpens the picture without changing it. Temporal questions (3/3 recall everywhere, all current/historical answers correct) and the stale-summary trap are solved by every condition β€” the baseline’s reader handles supersession from raw passages, and nothing in the graph improves or degrades it. The relational case the baseline misses thinly (q-rel-influences) is answered with explicit mechanism by both Basic and Local β€” the run’s one genuine point for stored structure composing across artifacts. The synthesis case Local misses (q-global-reliability) is genuinely vaguer than the baseline’s answer, naming performance issues without contention, benchmark, or incident specifics. Global mode’s synthesis territory went untested by design of the missing cells, not by verdict of the scores.

Graph structure is not automatically better retrieval

The honest summary of the local workload: strong conventional RAG is extremely difficult to beat, and the graph conditions do not beat it. After adjudication, every full-coverage condition answers eight or nine of nine decision cases correctly. The baseline does this with the smallest context (580 estimated tokens), the lowest latency, no indexing phase, and no stored state to maintain or correct. Basic matches the baseline’s quality at roughly 2.3Γ— the context and 7Γ— the latency plus a 999-second index build. Local costs roughly 12Γ— the latency for one genuine miss more. Global costs roughly 25Γ— with incomplete coverage.

This is a good result, not a failed experiment. It is the result the book’s discipline demands: the rival was built strong, the comparison was fair on the reader and the budget, and the new mechanism was not permitted to win by facing a weakened opponent or by hiding its costs. A chapter that had forced GraphRAG to win here would have taught less than this one does.

Where structure may still help

Three places remain open, each narrower than a general quality claim.

First, the relational case above: composing influences across five artifacts is the shape of question stored relationships exist for, and both graph conditions gave the mechanism explicitly where the baseline named identifiers. One case proves nothing; a relational family with ledger-derived scoring would prove something, and building it is a recorded obligation.

Second, corpus-wide synthesis is untested, not refuted. Global Search ran none of the synthesis questions. Its value proposition β€” precomputed community summaries when no passage is the answer β€” cannot be evaluated on a fixture whose questions a six-passage reader already answers. The small global-query family the design calls for does not exist yet; the chapter specifies it (a handful of theme-level questions, ledger-derived where possible, frozen-judge multi-dimensional scoring where synthesis requires judgement) rather than pretending the current suite covers it.

Third, downstream consumption is real and already committed. Chapter 5’s propagation runs over the derived graph through the snapshot adapter; Chapter 7 maps derived graph artifacts into lineage; Chapter 6 routes to graph modes selectively. None of that is evidence that the graph answers questions better. It is evidence that persistent structure has option value β€” a distinct claim the verdict separates explicitly.

Persistent mistakes

Book result. Persistent understanding creates reusable intelligence and reusable mistakes. The frozen index exhibits both: complete source mapping alongside a one-letter person split (J. LINDQVIST / J. LINQVIST, the typo entity owning three relationships), a spelling-split system (EVENT-STORE / EVENT STORE as distinct nodes of different kinds), and a spacing-split participant (A. NOVAK / A.NOVAK). A reader’s misreading disappears with the query; these misreadings are stored state, inherited by every later mechanism until a rebuild corrects them.

Three properties make stored errors structurally different from transient ones. They are silent: nothing in a later answer marks which parts came from a typo node. They are load-bearing: association, routing, and lineage all consume the graph as ground. And they are expensive to fix: correction means re-extraction and community rebuild, not a better prompt. The missing community-2 report is the same lesson in miniature β€” a generation failure persisted as structural absence, detected only because the health check names it.

The mitigation the architecture actually implements is boundary, not prevention. Raw sources stay canonical and reachable, so any derived claim can be re-derived or distrusted; provenance mapping exposes orphans before they are used as evidence; version metadata (corpus hash, package version, models, prompts, chunking, community settings, timestamp, code commit) identifies each derived build so errors attach to versions rather than to history itself. What the architecture does not implement is verification of derived content β€” that is Chapter 7’s work, and this chapter’s errors are its motivating exhibits.

Keep the original history

The rebuild rule follows from the epistemic status: source corpus canonical, derived graph rebuildable, versions explicit. Changing the GraphRAG version, prompts, extraction model, embedding model, or source corpus creates a new derived-memory version; experimental state is never silently overwritten. The current backend records corpus version and hash, package version, chat and embedding models, chunking, community level, claim-extraction settings, and prompt hashes in every run manifest. Incremental update β€” what must be rebuilt when history grows, what happens to communities and provenance, what the version identity becomes β€” is documented as a requirement the current backend meets only by full rebuild, with LightRAG’s incremental union design (Guo et al., EMNLP Findings 2025) recorded as the contrast that makes the cost visible. Update economics, not update machinery, is this chapter’s finding.

Fallback runs downward through cost: graph-derived memory, uncertain or insufficient, falls back to strong Chapter 3 retrieval over raw sources. Chapter 6 will show derived mechanisms adding harm on some tasks; the fallback is what makes that finding survivable rather than fatal.

What did GraphRAG actually buy us?

Costs, measured where measurable, estimated nowhere. Index build: 999 seconds wall, 55 LLM responses, roughly 128,000 tokens for 20 small documents β€” indexing amortises only over query volumes this fixture never approaches. Query latency: Basic ~51 s, Local ~93 s, Global ~196 s mean with a 585 s maximum, against a baseline reader-side cost near 7 s. Context: 2.2–2.6Γ— the baseline’s tokens for equal-or-indistinguishable decision quality. Storage: a full parquet index plus embedding stores for edge lists a flat file could hold. Model calls per graph query: uncounted by the package’s public API and recorded as unavailable, not estimated.

Against that price, measured gains on direct question answering are zero in general: adjudicated decision accuracy is indistinguishable across full-coverage conditions, temporal handling is solved everywhere, and the one relational bright spot is a single case. Genuine option value exists downstream β€” traversal, selective routing, lineage mapping all consume the graph β€” but architectural reuse is not empirical quality, and the chapter does not convert it into a score.

Did persistent understanding earn its place?

Developmental verdict: Type C with a Type B core. On direct question answering over the current local workload, strong RAG remains sufficient and the graph adds quality nothing at substantial cost (Type B). The graph nevertheless supplies a persistent substrate β€” entities, relationships, claims, communities with complete source mapping β€” that associative retrieval, selective routing, and evidence lineage genuinely consume (Type C). No evidence supports Type A (core-role quality win) or Type D (net harm warranting demotion); Local’s vague synthesis answer and the stored entity errors are costs, not disqualifiers, contained by fallback and rebuild. The verdict upgrades to a book result when the recorded obligations are met: repaired scorer normalisation, a relational question family, a tested synthesis family, and cross-query consistency measurement.

The principle that survives any single package:

The transition from retrieval to memory is not the moment a language model first understands the past. It is the moment some interpretation of that past becomes persistent state capable of influencing future remembering.

Whether preserving that interpretation is worth what it costs is then an empirical question per workload β€” asked here, answered conditionally, and re-asked by every later layer that inherits the graph.

The next question

A graph is static. Its relationships sit until a query arrives, and retrieval over them β€” even entity-centred, even community-aware β€” is still lookup: compare the question against stored items and return the nearest. But remembering, in the systems this book takes as hypotheses rather than models, does something else. A cue touches one memory, activation moves along relations, and the needed memory arrives three steps later by a route no similarity computation planned. The map exists now, with its virtues and its typo nodes. The next chapter asks how recall should move through it β€” and whether moving buys anything that lookup, however well built, cannot.

References