Parsing, Chunking, and Retrieval Units

A search result tells the reader to hold a blue button for ten seconds. What it does not return is the heading above the instruction: Handheld R17 only. Someone asking about a router could see a plausible reset procedure while missing the restriction that makes it inapplicable.

Both pieces are present in the source. The retrieval unit contains only one. Before changing the ranking algorithm, we need to examine the boundaries of the material we made searchable.

The First Search Baseline evaluated exact passage identities. Changing where we split the text changes those identities, even when the source documents stay fixed. We cannot simply transfer the old relevance labels to the new fragments: we first need a rule for deciding whether each fragment, or combination of fragments, still supplies the required evidence.

Preserve the source before interpreting it

We extend the lexical baseline’s fictional collection with operations notes containing scoped procedures, a misleadingly similar procedure, fenced commands, a table and a deliberately long section. The existing specification, blog and laboratory records remain unchanged, and the earlier evidence retains its original definitions. Only the additional source material and the way it is divided into retrieval units are new here.

The parser works on a declared raw representation: UTF-8 text with LF line endings. Every retrieval unit refers back to an exact half-open span of that text. Its offsets count Python string codepoints, not bytes or browser positions; the start is included and the end excluded. The unit retains source, document, version, representation, segmentation and passage identities, together with its parent-content digest and span. Its returned text must equal that exact slice of the parent.

An exact span is meaningful only relative to a particular text representation. The W3C annotation model’s TextPositionSelector makes this dependence explicit and notes how edits can invalidate positions. In our contract, an edited body gets a new version and new references. We do not search for similar words and silently attach an old judgement to a revised passage.

examples/evidence_units.py recognises only the structures needed for this experiment: headings, paragraphs, triple-backtick code fences and contiguous pipe-table lines. It keeps internal blank lines inside fenced commands and retains each table’s header, separator and rows together. An unclosed fence fails explicitly rather than producing a guessed boundary. This is a deliberately limited teaching parser, not a general Markdown, HTML or PDF extractor. CommonMark’s fenced-block specification informs the treatment of fenced content; pipe tables follow our local convention.

The collection selects permitted, current source versions before parsing. Only those bodies are segmented. Otherwise, excluded text could still affect a public heading, token count, retrieval unit or embedding even if the eventual result list concealed it. This ordering is part of the experiment’s trusted-view boundary, not a general authentication system.

Judge evidence independently of units

The scoped-reset information need requires two pieces of source evidence: the device restriction and the complete instruction. We record them as two anchors in the same source version:

Each anchor also fixes the expected quotation and parent digest. Validation checks its complete source and version identity, span bounds and exact text. Matching words in a blog, another document or another version do not satisfy an anchor tied to this particular source.

An information need can specify alternative sufficient bundles of anchors. To satisfy one bundle, the selected material must cover every anchor in it; a different accepted bundle may provide another valid route. These alternatives are authored judgements, not relationships inferred from identical text. The lamp-status need accepts several declared occurrences, while the laboratory-key need accepts the laboratory statement but not the conflicting operations note.

We measure two things at a fixed cutoff:

1. Single-unit sufficiency: one selected unit contains every required anchor of an accepted bundle.
2. Bundle coverage: the union of selected source intervals covers every required anchor of an accepted bundle.

Coverage requires each entire anchor span, not a partial overlap. To test a bundle, the evaluator combines selected intervals only when they belong to the same qualified parent, merges their overlaps and checks for gaps. Repeating the same fragment cannot fill a missing portion. Several selected units may collectively cover a bundle even when no individual unit contains all its anchors.

Consider a heading that limits a command to one device. A block-based retriever might return the heading and command separately. Together they can cover the declared source evidence, although neither block is sufficient alone. Whether a later consumer combines them and obeys the restriction is a different question. This chapter measures potential evidence coverage, not consumer assembly, reliance or correctness.

When a need has no sufficient bundle in the declared collection, its coverage quantities are undefined and reported separately. A plausible returned passage cannot create supporting evidence by itself. All judgements were authored before these comparisons but remained visible to the same author, so they are not independently blind labels.

The protocol-held-out file contains eight rewordings of development questions. Seven reuse the very same non-empty anchor bundles; the eighth repeats the no-answer condition with an empty bundle. We call this the reworded-query slice because it tests wording changes for already constructed evidence needs, not new source material. Its recorded protocol separation is real, but its evidence is not independent: it cannot establish generalisation to unseen documents, new needs or independently authored questions. Agreement between the slices is therefore not two independent confirmations. The original split identifiers remain in the evidence records.

Change one boundary policy

We hold the eligible documents fixed and vary only how they are divided into searchable units. The experiment implements three deterministic segmentation schemes:

The overlapping windows can cut across paragraph, code and table boundaries; we include them to expose that trade-off, not to recommend such splitting. Sections keep neighbouring source context together, but may be too long for an encoder to represent fully. None of the schemes silently adds a heading to a fragment. Doing so would create a different searchable representation that must be declared and evaluated separately.

Every segmentation scheme assigns its own passage identities and manifest. The manifest maps each unit back to its exact retained source slice; the earlier paragraph-only resolver is not assumed to resolve arbitrary chunks. Tests check exact reconstruction, wrong source or version, partial anchors, interval gaps, duplicated intervals, split evidence and explicitly allowed alternative bundles.

To inspect an executed comparison from the repository root:

python -B experiments/part2-semantic/run.py --query s-d01

This command inspects the frozen query trace. It does not perform fresh retrieval or call the embedding model. The experiment documentation provides the implementation, original capture and separate fresh-encoding commands; reviewing the record must not be mistaken for generating a new result.

What actually changed

The shared source-bundle comparison, recorded in research/evidence-register.md, applies unchanged BM25 to each segmentation scheme. It uses ten development questions (nine with sufficient bundles) and eight reworded questions (seven with sufficient bundles). At a cutoff of three retrieved units, the recorded source-evidence outcomes are:

The scoped-reset question shows why these metrics are different. With blocks, BM25 selects a neighbouring procedure, the correct instruction and its scope heading. The selected units cover the required bundle together, but none alone is sufficient. With sections, the correctly scoped procedure is sufficient in the first result. With overlapping windows, the selected fragments omit part of the required instruction, so even their union does not cover the bundle. A high score for a fragment cannot make the missing words appear.

The higher section counts require care. Segmentation changes unit lengths, BM25’s collection statistics and the amount of text allowed by a three-unit cutoff. To expose that budget difference, a separate returned-text audit counts characters in the units actually selected by BM25:

The audit sums the Python-codepoint lengths of up to three returned units per question, then averages across all ten development or eight reworded questions, including no-answer cases. A shorter list contributes only the text returned, and an empty list contributes zero. Text repeated across overlapping units is counted again. Citation metadata and added separators are excluded. The calculation is reproducible from retained unit text and selections and is registered separately from the retrieval experiment.

The comparison holds the number of retrieval units fixed, not the number of characters, model tokens or context bytes. Sections carry substantially more text, which mechanically gives them more opportunity to cover a required anchor. Their apparent advantage cannot therefore be assigned solely to better boundary placement or ranking. Equal-character or equal-token comparisons, and independently constructed evidence needs, remain unmeasured. Nor can these source-bundle results be directly compared with Part I’s passage Recall or MRR.

For these constructed needs, sections frequently preserve the heading and instruction together, and the three-unit limit lets them return considerably more text. But source preservation introduces another boundary: a long unit can retain the correct passage even when an encoder never sees that passage. Dense Retrieval without Re-teaching Embeddings asks what happens when the stored evidence and its searchable vector represent different amounts of the source.

References and further reading

Background and extensions for this chapter:

• Karpukhin et al. (2020). Dense Passage Retrieval for Open-Domain Question Answering. EMNLP. Uses passages as retrieval units in open-domain question answering.
• Liu et al. (2021). Dense Hierarchical Retrieval for Open-domain Question Answering. Findings of EMNLP. Examines document context and hierarchical structure around passage retrieval.
• Sarthi et al. (2024). RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval. arXiv preprint, v1. Extends segmentation into a generated summary tree, unlike the exact source spans used here.

Related chapters

• Representation Is Part of Context. Continues the distinction between source information and its representation.