Building an Inverted Index

We search for cache reset, then repeat the search. Our first retriever reads every eligible passage and tokenises it each time. The documents have not changed, but we do the same preparatory work again.

The scan made matching easy to inspect, but repeats work when queries change more often than documents. An inverted index prepares the term-to-passage relationships once, so each query can consult the relevant lists instead of rereading every passage. The improvement must preserve the evidence contract from Documents, Identity, and Provenance: faster discovery is no use if a result loses its source, document or version identity.

This chapter changes discovery. It does not yet provide a better reason for ranking one match above another.

Turn the relationship around

A scan walks through passages to ask, “Which query terms occur here?” An inverted index reverses the lookup: given a term, it identifies the passages in which that term occurs. The result is a dictionary of terms, each pointing to a list of passage records called postings.

Consider the fictional instruction:

To reset the cache, run cache-clear.

Our token sequence is:

position:  0   1     2   3     4   5     6
token:     to  reset the cache run cache clear

The cache posting points to this passage with positions [3, 5], so its term frequency is two. The reset posting points to the same passage at position [1], with frequency one. Each posting connects a term to a retrieval unit; its positions identify occurrences within that unit, not locations in the original file.

Established background: the IR textbook describes index construction as tokenisation followed by a dictionary and postings, with document frequency recording how many indexed items contain a term. Its construction section and positional-index section distinguish item identifiers from token positions.

Our indexed unit is a passage, even though the conventional name document frequency is retained. Here it counts passage rows containing a term, not distinct parent documents or independent authorities. Two passages from the same document can each contribute one count. This choice will matter when BM25 uses those frequencies in Chapter 04.

Keep preparation consistent with the scan

The implemented tokeniser retains the first retriever’s vocabulary rule but preserves repetitions:

def tokens(text):
    return re.findall(r"\w+", text.casefold())

The original scanner turns these words into a set and loses repetition counts. Index construction keeps their order and repeated occurrences, so it can record positions and term frequencies. Search queries still use distinct terms: typing cache twice does not double its score contribution.

Case folding removes some differences in letter case; the regular expression treats punctuation as a boundary between word tokens. These are explicit choices about what can match, not lossless transformations of the source. Python’s regular-expression documentation defines Unicode word characters, and its case-folding documentation describes caseless matching.

Book result: Part I token controls. The frozen run retains these counterexamples:

Accented characters remain, but the tokeniser does not perform canonical Unicode normalisation. The frozen control shows that precomposed café and a visually similar spelling with a combining accent yield different token lists. There is also no stemming or stop-word removal. Introducing any of those operations would change the searchable representation and require a separate comparison.

These matching differences do not change the retained source text, identities or character spans. In particular, token position 3 means the fourth token in the indexed sequence; it is not character offset 3 in the parent text. Positions could support a later phrase matcher, but this implementation does not accept or interpret phrase queries.

A posting points into a fixed manifest

A posting can store a compact integer row number, but that number only makes sense within a particular index. It cannot by itself identify the source passage that a reader may need to open later.

The index begins with Chapter 02’s eligible passage packets, sorted by their complete references. It copies them into a fixed manifest row table. Postings then store row ordinals, term frequencies and positions rather than repeating full references. Each manifest row retains the source, document, version, representation, segmentation, passage ID and expected content digest, together with the text, source span and provenance metadata needed to reconstruct the evidence.

The build loop is small:

for row_id, row in enumerate(rows):
    sequence = tokens(row["text"])
    for position, term in enumerate(sequence):
        postings[term].setdefault(row_id, []).append(position)

examples/lexical_index.py implements this construction, including input validation, deterministic ordering and public output projection. Each row ordinal belongs to a fingerprinted index. Rebuilding another collection view may assign the same passage a different row number. A saved reference such as row=4 is therefore incomplete without the identity of the index and its manifest.

Given a row number, index.reference(row_id) reconstructs the complete Chapter 02 reference from the manifest. The collection can then resolve that reference under the applicable view. The passage hash is not decoded to recover the missing fields; the manifest supplies them. The frozen index.json keeps rows and postings together so this path from term to evidence can be inspected.

Identical wording does not justify merging evidence identities. The public specification and blog produce separate manifest rows and postings even though they contain the same reset sentence. One query can find both; their distinct references still tell us where each occurrence came from.

Retrieve candidates without rescanning their text

At query time, look up the posting list for each distinct query term and take the union of their row IDs. For cache reset, a passage containing either term becomes a candidate. This is an OR-style candidate rule, not a general Boolean query language: it does not require both terms or interpret an explicit query operator.

The indexed-overlap scorer counts distinct matched query terms, just as the original scanner does. It orders candidates by score, breaks ties by stable identity, then takes the requested prefix. The trace still distinguishes retrieved, eligible and selected. Here eligible mirrors the retrieved candidate IDs because the indexed collection was filtered before construction; that field is not evidence of a new permission check at query time.

Run the shared implementation from the repository root:

python -B examples/lexical_index.py --inspect cache
python -B examples/lexical_index.py "cache reset" --method scan
python -B examples/lexical_index.py "cache reset" --method overlap

The command-line example rebuilds the index each time, so its individual invocations do not demonstrate persistent reuse. To issue several queries against one prepared index, keep the index object in a Python process with examples/ on the import path:

from document_identity import Collection
from lexical_index import InvertedIndex

collection = Collection.load("examples/fixtures/part1-v1.json")
index = InvertedIndex.from_collection(collection, "baseline")
index.search("cache reset", method="overlap")
index.search("Q77", method="overlap")

Both calls consult the same rows and postings instead of re-tokenising the passage collection. The example has no background index service or saved-index loader; reuse lasts only while this Python object remains available.

Implementation status: implemented and executed. Book result: Part I index contract. The recorded baseline view contains thirteen eligible passages, indexed as 46 terms, 78 term–passage postings and 117 token occurrences. The cache reset query visits eight posting entries across two lists; the original scan examines thirteen passages. These are different units of work, so eight versus thirteen is not a latency or speedup ratio. The observation shows which repeated preparation and lookup work the index changes.

The retained canonical JSON index is 23,877 bytes, including copied text, provenance, positions and readable field names. This is a serialised teaching artefact, not a compressed-index or resident-memory measurement. Query latency, peak memory and throughput remain not measured. Index construction also costs work, so a small or frequently changing collection might not benefit from preparing one.

The supporting records are in experiments/part1-baseline/frozen/, with claim boundaries in research/evidence-register.md. On the frozen development queries, the original scanner and indexed-overlap search produce the same passage order and scores. Reversing input row order also produces byte-identical index data. These checks establish behavioural equivalence and determinism on this collection; they do not establish a general performance gain.

Eligibility is part of the indexed view

The constructor first calls Collection.public_passages(snapshot_id). Only passages permitted by Chapter 02’s trusted scope, lifecycle, freshness and version rules are then tokenised and indexed. The public postings, frequencies and collection statistics therefore exclude restricted records, not just restricted snippets in the final result.

The frozen negative control changes the fictional restricted text and finds the public index—including its statistics—byte-identical. This distinction matters for BM25: removing restricted hits from the displayed results after calculating document frequencies from them would produce a different scorer and potentially expose information through the statistics.

The two arrangements below differ only in where the eligibility gate sits. The book implements the first; the second is drawn to name the failure it would create, not as a measured comparison.

    flowchart LR
    subgraph build["Implemented: gate before indexing"]
        A["Named view: permitted passages only"] --> B["Tokenise and build postings"]
        B --> C["Collection statistics over the eligible view"]
        C --> D["Query: postings union, score, top-k"]
    end
    subgraph post["Avoided: gate after scoring"]
        E["All records tokenised and indexed"] --> F["Collection statistics over everything"]
        F --> G["Query: postings union, score, top-k"]
        G --> H["Remove restricted hits from the result"]
    end
    classDef warn stroke:#b3261e,fill:#fbe9e7;
    class H,F warn;
  

In the second arrangement the restricted text has already influenced the statistics and the candidate scores, and the removal happens after the ranking decision. Rebuilding the index for a different view therefore changes more than the visible rows.

Building a separate index for each permitted view is one architecture, not the only one. A correctly filtered shared index is another possibility. In either case, protected information must not reach public results or diagnostics, and eligibility must constrain candidates before final top-k selection.

An index records the eligibility decision made for its indexed view, not the permissions that may apply later. In the regression test, an older index still finds a passage subsequently restricted. Resolving that reference under the newer view refuses it; rebuilding the index for that view excludes it. The trusted caller remains responsible for choosing and maintaining the current view. An index fingerprint tells us what was indexed, not what may be disclosed now.

Rebuild rather than rewrite history

Updating this teaching index means rebuilding it from a different frozen collection view. There is no incremental database, write-ahead log or production deletion service. That deliberate limitation makes changes in identity and collection statistics easier to inspect.

The fixture follows the specification through a declared move, an edit and removal. Moving it changes its observed location without changing its exact passage reference. Editing it creates version v2 with a new reference. Removing it leaves no current reset candidates in that view. Chapter 02’s named-view resolver can still address retained history when permitted; current search never silently substitutes historical text.

Rebuilding recomputes postings and collection statistics together. An incremental implementation would need to maintain the same invariants when rows change or disappear, including updates to posting lists and document frequencies. That is a separate engineering problem, not an invisible feature of this small index.

Larger collections can also make index construction exceed available memory. Heinz and Zobel’s single-pass design accumulates compressed postings in memory, writes term-ordered runs when that memory fills, then merges those runs. “Single-pass” refers to reading the source stream; temporary files and merging remain necessary. We keep the simpler in-memory rebuild. A run-based version would still need manifest-qualified row references: compressing an ordinal cannot restore source identity that was discarded.

The new tests check postings and positions, complete reference reconstruction, duplicate-source survival, lifecycle rebuilding, tokenisation failures, eligibility and deterministic output. They run alongside the unchanged earlier tests:

python -B -m unittest discover -s tests -p test_lexical_index.py -v
python -B experiments/part1-baseline/run.py --verify experiments/part1-baseline/frozen

A candidate list is not a ranking argument

For cache reset, indexed overlap gives a verbose cache note and the short command passage the same score. The tie-breaker puts the verbose note first. That order is reproducible, but the note’s source identifier gives us no reason to consider it the better answer to the task.

We can now go from a query term to a posting, from a posting to a manifest row, and from that row to an exact source reference—without rescanning every passage for each query. But finding and identifying candidates does not tell us which is most useful. BM25: Ranking with Terms uses the retained frequencies and lengths to make that next ranking decision explicit.

References and further reading

Background and extensions for this chapter:

• Zobel and Moffat (2006). Inverted files for text search engines. ACM Computing Surveys. Provides background on dictionaries, postings and index construction.
• Heinz and Zobel (2003). Efficient single-pass index construction for text databases. Journal of the American Society for Information Science and Technology. Extends the simple rebuild towards index construction with bounded memory.
• Broder et al. (2003). Efficient query evaluation using a two-level retrieval process. CIKM. Shows how postings can screen candidates before complete scoring.

Related chapters

• From Similarity to Search. Continues from lexical indexing to the distinction between search policy and execution.