Documents, Identity, and Provenance

The sentence is correct. The attribution is wrong.

A fictional specification says, “R17 reset: hold the blue button for ten seconds.” An old blog repeats those words exactly. Suppose we key our document records only by a digest of their text and keep one metadata record per digest. The blog is inserted second, so its source metadata replaces the specification’s. The quotation survives unchanged; the distinction between its two origins disappears.

The chapter’s first executed counterexample reproduces that loss in fictional data; no real device or generated answer is involved. The stored sentence remains accurate to both source texts, but a later citation renderer cannot recover the two attributions once their records have been collapsed.

The Information Selection Problem kept identity fields alongside each search score. This chapter tests why they matter. When a document moves, which identity should remain stable? When its text changes, which identity must change? And when a reader follows an old citation, which version should that reference identify?

The right words can have different origins

The executed digest-only control in experiments/ch02-identity/run.py reduces to this operation:

hash_only = {
    version["content_sha256"]: version
    for version in duplicate_versions
}

duplicate_versions contains the specification followed by the blog. Because their retained text is identical, both records have the same dictionary key. Assigning the second value replaces the first: only the blog’s metadata survives. The hash calculation has behaved as designed; the error is treating a content digest as a unique source record.

A content digest identifies the hashed message content, not the publication that supplied it. Established background: NIST’s Secure Hash Standard publication describes digests as supporting detection of message changes. The same digest cannot, by itself, establish who published the text, what permissions govern it or which source is authoritative.

The repair is to keep a separate source-and-document identity for each record, even when its text matches another record. A storage system could still deduplicate the underlying bytes without merging attribution metadata. Our example makes only the identity change: it does not implement shared-blob storage.

Source identity is not source authority. In this fixture, toy:spec names a declared origin, while “fictional specification” is a separately supplied authority label. Neither verifies the publisher’s credentials or the truth of the instruction. Retrieval must retain both declarations without treating either as an independent trust verdict.

Book result: identity survives retrieval. In the supplemental both-eligible view, both public sources are declared current. A search for R17 reset with a limit of two retrieves and selects both passages, each with score 2. Their text is identical—“R17 reset: hold the blue button for ten seconds.”—but their source, document and passage identities remain different:

The full references and trace are in experiments/ch02-r1/frozen/observations.json; neither source record is overwritten. This is a supplemental view, not a change to the original experiment: there, the blog is declared stale and excluded from current retrieval. Staleness limits eligibility; it does not establish that the blog’s words are false.

A location cannot do every identity job

The next failure concerns location. The specification first appears at fictional/manual/reset.txt—its first recorded location in the fixture, not a verified original publication address. We distinguish three locators: an observed locator says where a named view places the document; a capture locator says where a particular version was captured; and a retained-storage locator identifies the passage in our retained material. Those locations serve different purposes, even if their values sometimes coincide.

The three are attributes of an identity, not identities: two of them may hold the same string at one moment and diverge later, and none of them is a fetchable address in this teaching model.

The fixture then declares a move to fictional/handbook/reset.txt, without changing the text. If the path is the document’s identity, one logical document now appears to be two different documents. The declared continuity is lost when the address becomes the key.

Next, the instruction at that path changes from ten seconds to twenty seconds. The locator remains stable, but the text does not. A store that retains only the latest text under the path cannot answer a historical citation’s question: did this reference identify the ten-second version or the twenty-second version?

The two failures require two identities. Document identity follows the same declared logical item through a move and an edit. Version identity identifies one recorded state of that item. Moving the unchanged document preserves its version; changing the instruction creates a new version under the same document identity.

The retriever does not infer this continuity from matching text or filenames. The fixture explicitly declares that the before-and-after observations belong to one document. Equal digests can confirm equal retained content, but they cannot independently establish that two observations have the same logical identity.

The diagram below separates what the fixture declares from what it merely makes equal. Follow the two labelled edges first; then look at the pairs with no edge, where an identity question has no declared answer.

    flowchart TD
    subgraph declared["Declared continuity: one document, three observations"]
        B["spec-before<br/>fictional/manual/reset.txt<br/>version v1"]
        M["spec-moved<br/>fictional/handbook/reset.txt<br/>version v1"]
        E["spec-edited<br/>fictional/handbook/reset.txt<br/>version v2"]
        B -->|"move"| M -->|"revision"| E
    end
    subgraph undeclared["Equal or similar, but not declared"]
        P1["toy:spec, version v1"]
        P2["toy:blog, version v1"]
        Q1["fictional/unknown/guide.txt"]
        Q2["fictional/unknown/guide-copy.txt"]
        P1 -.->|"identical words: continuity unknown"| P2
        Q1 -.->|"identical words: continuity unknown"| Q2
    end
    classDef unknown stroke:#8b95a1,stroke-dasharray:5 4,fill:#f0f2f5;
    class P1,P2,Q1,Q2 unknown;
  

The node text makes the two failures visible at once. spec-before and spec-moved share version v1 while their locators differ, so a path cannot be the version identity. spec-moved and spec-edited share a locator while their versions differ, so a path cannot be the version identity either. Only the declared edges connect them.

Now consider observations at guide.txt and guide-copy.txt. They contain identical reset text but have no assigned canonical document identity. A supplemental regression supplies a second pair: two identified records with equal text but no declared relationship. Both pairs return unknown, because the function finds no direct identity declaration. Removing a declared move link likewise removes that direct relationship. Identical words alone do not justify a continuity claim.

unknown is an absence of a direct declaration, not proof of different documents. For example, the specification’s initial and edited observations belong to one declared logical document, connected by a move and then a revision. Nevertheless, continuity("spec-before", "spec-edited") returns unknown: the function examines direct edges, not multi-step paths. It also returns unknown for unrecognised observation identifiers, so this lookup cannot distinguish a missing record from a known record with no direct edge.

The implemented mechanism preserves supplied identities and checks direct declarations; it does not resolve document identity automatically. The digest-only and locator-only controls are constructed counterexamples, not evidence of how often such failures occur in real repositories. Nor does successfully preserving a declared relationship prove that the declaration itself is correct.

The minimum model the failures require

Our book contract separates the following objects:

Locators, digests, scope, freshness and timestamps are attributes of these identities, not substitutes for them. A publication timestamp and a capture timestamp describe different events; unknown publication time must remain unknown. Likewise, a stale record can still contain accurate words, and retaining an old version does not make it eligible for a current request.

Established background: the W3C PROV data model distinguishes entities, activities and responsibility, and defines revision as a derivation relation. We use that distinction to explain recorded origin and transformation. The small tuple model here is our design choice; it is not a PROV implementation or conformance claim.

Buneman, Khanna and Tan distinguish where-provenance, the origin of an output value, from why-provenance, the input data contributing to an output under a query. Our exact source span answers an origin question. A record of the transformations and selections would answer a different dependency question. For example, a hypothetical ranker might return one source’s sentence because other documents changed its collection statistics: those documents can affect selection without becoming the sentence’s origin. This analogy does not make our manifest an implementation of the paper’s formal database model, and neither form of provenance proves that a later model used the sentence correctly.

The logical keys are deliberately inspectable:

document key = (source, document)
version key  = (source, document, version)
passage key  = (version key, representation, segmentation, start, end)

The representation identifies the exact text in which character offsets are measured; segmentation identifies the rule that divided that text into passages. The passage ID is a hash derived from the complete key. It is not a reversible encoding of the key, so the evidence record must retain the original fields needed to reconstruct an exact reference.

An exact reference also includes the expected digest of its parent text. If the retained text changes while the version label stays the same, resolution fails rather than treating the changed text as the cited version. This is an integrity check, not authentication; the digest cannot replace the source, document or version identity.

A passage belongs to a particular parent

The specification contains a reset instruction and two occurrences of the paragraph “Check the status lamp.” The two paragraphs have identical words but occupy different positions in the same parent version. Their recorded spans are [50, 72) and [74, 96): each includes its start offset and excludes its end offset. Both resolve to the expected text, while their different positions produce different passage identities.

The retained representation is UTF-8 text with LF line endings, interpreted as an unmodified Python string. Its spans count Unicode code points—not UTF-8 bytes, word tokens or visual columns. Search may case-fold or tokenise a separate representation, but a source reference must still slice the original retained text at its recorded offsets.

Established background: the Web Annotation Data Model’s text-position selector records ranges in a representation and warns that edits can make them brittle. Its selection and normalisation rules are richer than this fixture’s raw-text contract. We do not claim general selector conformance or automatic quote re-anchoring.

A historical reference names the version it was created for, not whatever text now occupies the same path. Resolution returns unavailable if that version cannot be accessed or if the passage ID, representation or expected content digest fails validation. It never silently substitutes similar wording from another source or version. A plausible replacement would be a different reference, not a successful resolution of the old one.

Even if a paragraph’s words survive an edit elsewhere in the parent document, its old reference still identifies the old version. Recognising continuity between passages across versions is a separate problem; this example neither implements nor evaluates that mapping.

Established background: choosing a retrieval unit is separate from choosing a physical file. Manning, Raghavan and Schütze discuss that choice in Introduction to Information Retrieval, §2.1.2. We use blank-line paragraphs to demonstrate identity and parent mapping, not to claim an optimal chunk size. Controlled chunking comparisons belong later.

Run the identity experiment

Implementation status: implemented and executed. The standard-library module examples/document_identity.py builds an offline manifest, constructs exact references, resolves retained historical versions and produces a permitted current view. Its retrieval adapter still calls Chapter 01’s unchanged overlap scanner: the new behaviour concerns identity and eligibility, not a new ranking algorithm.

Run current retrieval from the repository root:

python -B examples/document_identity.py "R17 reset" --snapshot revised
python -B examples/document_identity.py "R17 reset" --snapshot removed

The first command selects the recorded v2 reset passage. The second returns no candidates under this fixture’s current-view policy. Neither command generates an answer.

The following code uses the offline fixture helper to create a reference to the earlier version, then asks for its history in the revised view:

from document_identity import Collection

collection = Collection.load()
reference = collection.reference("toy:spec", "reset-guide", "v1")
result = collection.resolve_history(reference, "revised")
print(result["record"]["text"])

With examples/ on the Python import path, this prints the original ten-second instruction. The returned historical record is marked superseded and ineligible for current retrieval. In contrast, resolve_current refuses the old reference in the revised view. The two methods answer different questions: what did the citation identify, and may that version be retrieved now?

To inspect the supplemental two-source retrieval trace with the same import setup:

collection = Collection.load("examples/fixtures/ch02-r1-v1.json")
trace = collection.retrieve("R17 reset", "both-eligible", limit=2)
for record in trace["selected"]:
    print(record["source_id"], record["passage_id"], record["text"])

This prints the blog before the specification. Their scores tie, so the unchanged Chapter 01 scanner applies its deterministic identity ordering. Both references survive selection; the shared sentence has not caused their sources to be merged.

Book result: Chapter 02 identity lifecycle. The frozen development experiment records these outcomes:

The fixture, configuration, offline manifest, observations, environment, exact commands, input hashes and outputs are retained in experiments/ch02-identity/frozen/. The narrow claims are recorded in research/evidence-register.md. The implementation preserves the fixture’s declarations; the experiment does not independently verify the fictional source relationships.

To verify the package and replay it without overwriting evidence:

python -B experiments/ch02-identity/run.py --verify experiments/ch02-identity/frozen
python -B experiments/ch02-identity/run.py --output .artifacts/ch02-identity-replay

Choose a fresh replay directory. Compare observations.json and manifest-data.json byte for byte; execution times and test durations may differ. The manifest data is an offline inventory that includes fictional restricted records. It is not the public retrieval response.

The supplemental retrieval, permission-view and direct-continuity observations have a separate package; the original evidence remains unchanged:

python -B experiments/ch02-r1/run.py --verify experiments/ch02-r1/frozen
python -B experiments/ch02-r1/run.py --output .artifacts/ch02-r1-replay

Use a fresh destination for this supplemental replay too. Compare its observation and manifest-data files with the corresponding frozen files. The additional observations and four regression tests remain registered separately under RET-CH02-R1; they do not alter the original identity experiment.

Retained does not mean currently retrievable

The fixture defines four collection views: initial, moved, revised and removed. For current retrieval, a passage must belong to the selected current version and satisfy the view’s declared public scope, active lifecycle and freshness checks. These checks run before its text reaches the scanner. They are the rules of this experiment, not a universal definition of freshness or a live authorisation service.

In the removed view, the specification is no longer eligible for current retrieval, and resolve_current refuses its stale reference. The separate blog record still exists, but this view also declares it stale. Consequently, the removed-view query returns no candidates. That absence follows from this view’s eligibility rules; removing one source would neither revoke nor authorise a different permitted copy.

Retaining v1 means its text can still be resolved historically when the selected view permits access. If that view restricts the document, resolution returns unavailable instead. Retention protects the possibility of historical attribution; it does not grant perpetual access.

A reference’s historical identity is fixed; permission to reveal its text depends on the chosen view. Book result: caller-selected permission view. In the supplement, the same old reference resolves under the public revised view but returns unavailable under later-restricted. Selecting revised again reveals the original words, even after a more restrictive view has been defined. The resolver checks the named view, not the latest effective policy. This is why a historically valid reference cannot itself authorise historical disclosure.

In this teaching contract, a trusted caller chooses the snapshot. The prototype neither authenticates the caller nor independently looks up the latest permissions. Scope applies across the document’s versions within the selected view, but enforcing up-to-date permission for historical access remains unimplemented. The raw scanner, offline manifest and fixture-reference helper are not protected services. Missing and restricted references return the same outward failure, avoiding a reason-specific disclosure while leaving the larger authorisation problem unresolved.

The packet’s locator is an identity-derived retained-storage label, toy:retained/…, not a live source URL or a storage service. It retains capture_locator separately from observed_locator and observed_locator_version. A path observed for v2 cannot establish where to fetch historical v1; the resolver serves retained text rather than consulting a web archive or repairing URLs.

What has been established

The original test run passes nineteen Chapter 02 tests and the existing twenty-one Chapter 01 tests. The supplement adds four passing regression tests for two-source retrieval, differently scoped views, identified equal-text records without an edge, and indirect/unknown-identifier lookup. The original checks cover declared identity transformations, parent slices, deterministic manifest rebuild, unavailable references and scope/lifecycle boundaries. The Chapter 01 manuscript, implementation and frozen evidence remain unchanged.

These tests establish behaviour on the constructed collection, not general entity-resolution accuracy, production security, verified source authority, optimal passage size or better real-world citations. The declared identities and relationships may themselves be wrong; content digests are not signatures, and a manifest cannot restore text that was never retained. Nothing in these fixtures estimates how often identity failures arise elsewhere.

The evidence chain remains:

available information -> retrieved -> eligible -> selected -> admitted -> used -> helpful

Within the declared collection, we can now trace selected candidates through changes of location, text and eligibility. That establishes reference behaviour, not what happens after retrieval: context admission, actual evidence use and downstream helpfulness remain not measured. Embedding geometry, memory retention and context assembly are separate responsibilities.

We can now answer three questions that a score alone could not: which exact passage was found, which recorded version it belongs to, and whether the chosen view permits it to be returned. Building an Inverted Index preserves those references while changing candidate discovery. How can terms point to stable passage identities without rescanning every passage for every request?

A posting in that index can hold a row identifier rather than copying every provenance field. The referenced immutable manifest row must still reconstruct the full source, document, version, representation, segmentation, passage ID and expected content digest. A passage ID alone cannot stand in for this contract. Whether the implementation uses one index per permitted view or a correctly filtered shared index, current eligibility must be checked before protected candidates, snippets or traces are exposed and before final top-k selection. Chapter 03 explores one concrete design without claiming it is the only safe architecture.

References and further reading

Background and extensions for this chapter:

• Buneman, Khanna and Tan (2001). Why and Where: A Characterization of Data Provenance. ICDT. Distinguishes the origin of data from the transformations that produced it.
• Moreau et al. (2011). The Open Provenance Model core specification (v1.1). Future Generation Computer Systems. Extends provenance into explicit artefact, activity and dependency records.
• Sandve et al. (2013). Ten Simple Rules for Reproducible Computational Research. PLOS Computational Biology. Supports retaining exact inputs, versions and intermediate artefacts for replay.

Related chapters

• Advanced Agents From First Principles 25: Can You Reproduce an Agent Run Months Later? Add Deterministic Replay and Provenance. Extends exact source identity into replay manifests and execution provenance.