BM25: Ranking with Terms

Two passages match cache reset. One is a long maintenance note that mentions the topic repeatedly. The other gives the command the user needs:

To reset the cache, run cache-clear.

Both passages contain the two distinct query terms, so our overlap scorer ties them. The identity-based tie-break puts the verbose note first. That order is reproducible, but nothing about the tie-break establishes that the note better satisfies the request.

Building an Inverted Index gave us candidate passages, term frequencies and lengths without repeatedly scanning their text. Now we need a ranking rule: when passages share query terms, what should make one outrank another? BM25 uses those retained statistics to answer that limited question. It does not determine whether a passage is true, permitted or sufficient for the task.

Counting more is not enough

Counting every matching occurrence would distinguish a passage that mentions cache repeatedly from one that mentions it once. But under raw frequency counting, each repetition adds just as much as the last, even when it contributes no new instruction.

The verbose note in our constructed example has more matching occurrences than the command passage. A useful ranker may give repeated terms some weight without treating the seventh occurrence as equally informative as the first. This motivates frequency saturation: additional matches still count, but their marginal contribution decreases.

We also need to distinguish common vocabulary from discriminating terms. If nearly every eligible passage contains cache, another occurrence tells us relatively little about which passage the user wants. A rare identifier may narrow the possibilities much more. Inverse document frequency (IDF) assigns greater weight to terms that occur in fewer indexed passages.

Finally, two occurrences in a seven-token command and two occurrences in a lengthy maintenance note need not mean the same thing. Length normalisation changes the weight of a match relative to the indexed passage’s length. It is an assumption about where useful evidence tends to be concentrated, not a rule that short passages are inherently better: a long passage may contain the only complete answer.

BM25 brings these three ideas together: how often a query term occurs, how rare it is across the eligible collection, and how passage length affects the value of repetition. We can now specify the precise variant we will execute.

State the variant before showing a score

BM25 names a family of closely related scoring formulations, not one universally identical implementation. Established background: Robertson and Zaragoza’s The Probabilistic Relevance Framework: BM25 and Beyond, §3.4.5–3.5.1, develops frequency saturation and soft length normalisation and discusses variations. To reproduce a numerical score, we must state which formulation, collection unit and parameters we chose.

Let d be an eligible passage and t a distinct query term. The quantities below are all defined over eligible passage rows in the indexed view, not over source documents:

The formula is:

idf(t) = ln(1 + (N - df(t) + 0.5) / (df(t) + 0.5))

K(d) = k1 * (1 - b + b * L(d) / avgL)

score(q,d) = sum over distinct matching query terms t:
                 idf(t) * f(t,d) * (k1 + 1) / (f(t,d) + K(d))

Here ln is the natural logarithm. We use a positive, smoothed IDF and the (k1+1) frequency numerator. The sum considers each distinct query term once, so repeating a word in the query does not increase its weight. We fix k1=1.2 and b=0.75 before comparison; neither parameter was tuned on the observed results.

Lucene’s BM25Similarity API documents this positive IDF expression and the same default parameter values. That makes the choices recognisable, not the implementations interchangeable. Our Python scorer defines its own passage lengths, collection statistics, numerator and treatment of token-empty rows; it does not reproduce Lucene’s norm encoding, sparse-field behaviour or exact scores.

In a classic unsmoothed log-odds form, a term appearing in much of the collection can receive a negative IDF. Our ln(1 + …) form keeps IDF positive for terms present in the eligible collection. This is a scoring choice, not an algebraic rewriting of the classic expression. It means a matched term adds a positive contribution, however common it is.

The table lists the variant choices a reproducer would have to match. Each one is a decision this chapter makes explicitly; none of them is forced by the name BM25.

Lucene’s defaults agree with the first two rows, which is why the implementation is recognisable. The remaining rows are ours: the same formula with different collection units and parameters is a different experiment.

Read the saturation rule

To see saturation, hold the passage length and collection statistics fixed while increasing the matched term’s frequency f. The numerator grows with f, but so does the denominator. The first few occurrences matter more than later ones; the frequency factor approaches k1+1 instead of increasing without limit. This is the part of BM25 that stops raw repetition from dominating by itself.

k1 controls how much repetition can change a matched term’s contribution. With k1=0, one occurrence and many occurrences contribute the same IDF; larger values let additional occurrences matter over a wider range. A term absent from the passage still contributes zero. Saturation only applies after a term has matched.

b controls the length component of K(d). At b=0, a passage’s length does not affect its term contribution. At b=1, the adjustment uses the full ratio of passage length to average length. With our intermediate b=0.75, the same term frequency contributes less in a longer-than-average passage and more in a shorter one, all else equal.

The two panels below evaluate the stated formula at the declared parameters. They are a mathematical illustration, not a measurement: the horizontal axis is a hypothetical occurrence count, not an observation from the fixture.

The numerator factor is common across candidates for a fixed parameter setting. Robertson and Zaragoza describe it as a common scaling variation that does not change the resulting ranking. It still matters when reproducing numerical scores. Omitting it while copying a worked example would produce a different number even if candidate order stayed the same.

BM25 is making an explicit bet: term frequency, collection rarity and length help predict useful lexical matches. It does not interpret the passage’s claim. A warning that no procedure exists may match the query more strongly than the procedure itself, and a high score cannot authenticate a source or grant access to it.

Work one passage all the way through

Book result: Part I worked BM25 calculation. We can now calculate the command passage’s score without treating BM25 as a black box. The frozen baseline has thirteen eligible passage rows containing 117 indexing tokens, so the average length is avgL=9. The command passage contributes seven tokens:

to reset the cache run cache clear

For query cache reset, the retained inputs are:

The length component is:

K = 1.2 * (0.25 + 0.75 * 7/9) = 1

Thus the two term contributions are:

cache: ln(4)     * (2 * 2.2)/(2 + 1) = 2.033232
reset: ln(28/11) * (1 * 2.2)/(1 + 1) = 1.027740
total:                                  3.060972

The two positive contributions add to 3.060972, the command passage’s BM25 score for this query and indexed view. Displayed values are rounded; the runner retains full floating-point inputs and contributions and checks an independent calculation against the implementation. The worked_calculation record in experiments/part1-baseline/frozen/observations.json makes the arithmetic inspectable.

Notice that the score also depends on which passages we counted. Only eligible rows enter N, df and avgL. The public specification and blog have identical wording but separate passage identities, so both affect the statistics. That follows from our passage-level collection definition; it does not make duplicate wording independent authority. Whether to group duplicate content is a separate retrieval-policy choice.

Consequently, the same passage and query can receive a different BM25 score when the eligible collection changes. A revision, exclusion or inclusion may change N, document frequencies or avgL without changing that passage’s text. A reported score belongs to its indexed collection view; it is not a universal property of the passage.

Compare the competing explanations

Why might length normalisation help in one case and hurt in another? Robertson and Zaragoza distinguish verbosity—more words about essentially the same subject—from scope—more words because a document covers additional subjects. Penalising repetition may be useful for verbosity, but a longer passage may be necessary when the answer genuinely spans several topics. BM25’s soft length adjustment cannot tell those cases apart. Our b=0.75 is a declared compromise, not a measured diagnosis of why these particular passages are long.

To see what each scoring choice changes, the frozen run compares the same cache candidates under simple controls. Distinct-term overlap counts matched query terms once. Raw term count includes every occurrence. The displayed TF–IDF control multiplies raw frequency by our positive IDF, with no saturation or length adjustment. The Boolean control only checks presence; it does not rank relevance. These are deliberately specified controls, not every possible implementation of TF–IDF or Boolean retrieval.

Book result: Part I cache ranking controls.

For this deliberately constructed request, BM25 ranks the judged command above the verbose note while the displayed overlap and frequency controls do not. That is a real result for the frozen fixture and an author-specified judgement, not independent evidence that BM25 generally improves retrieval. It also says nothing about whether the fictional command is factually correct. Chapter 05 will test the ranking rules across more than this motivating pair.

Both overlap and BM25 rank the same OR-generated candidate pool. BM25 changes their order, not whether a passage was found. The paraphrase restore factory settings still shares no indexed term with the reset instruction, so neither lexical method retrieves it. A better ranking function cannot promote evidence that candidate generation never supplied.

Nor does BM25 always put the judged instruction first. For cache cache, the recorded development trace ranks an archive passage above the command. The archive discusses the topic without supplying the requested instruction. Term statistics can improve an ordering in one case and mislead it in another; they cannot determine the user’s information need.

Inspect the implementation

Implementation status: implemented and executed. examples/lexical_index.py separates candidate collection from term contributions. The central calculation is:

idf = math.log1p((n - df + 0.5) / (df + 0.5))
return idf * f * (k1 + 1) / (
    f + k1 * (1 - b + b * length / average)
)

Run:

python -B examples/lexical_index.py "cache reset" --method bm25
python -B examples/lexical_index.py "cache reset" --method overlap
python -B examples/lexical_index.py --inspect reset

Inspect the returned candidate trace rather than only its final rank. For each matched term, it retains frequency, document frequency and contribution; the original source, version and passage reference remain attached. Candidates sort by descending score, with stable reference identity breaking ties. Neither a tied score nor a large score supplies an ambiguity verdict or calibrated confidence.

Empty queries and punctuation-only queries return no candidates. A collection with no eligible passages has no scores to calculate. A token-empty passage contributes no term posting; if present as a row, it still counts in this implementation’s declared N and average-length denominator. A wholly empty source body produces no paragraphs under Chapter 02’s segmentation rule.

The tests check the numerical formula, saturation, rarity, zero frequency, length behaviour at b=0 and b=1, and the k1=0 boundary. They also reject non-finite or negative k1, out-of-range b and inconsistent counts. A matched passage contains at least one indexing token, so its eligible view has a positive average length; the scorer does not need to divide by zero for an actual match.

The implementation remains a teaching system. Trusted view selection supplies eligibility; BM25 does not enforce current permission changes, authenticate metadata or make the raw underlying scanner a protected service.

A plausible ranking still needs a comparison

We have gained an inspectable ranking rule: the focused command outranks the verbose note for cache reset, and every term contribution can be reproduced. We have not solved lexical mismatch or intent. The paraphrase remains undiscovered, and the archive still outranks the judged command for another query.

The cache example taught us how BM25 works; it cannot independently establish that BM25 works better. The First Search Baseline therefore fixes information needs, judgements and query splits before comparing the original scan, indexed overlap and BM25. We will report both improvements and non-improvements rather than promote one constructed example into a general result. Context admission, evidence use and downstream helpfulness remain not measured.

References and further reading

Background and extensions for this chapter:

• Robertson and Zaragoza (2009). The Probabilistic Relevance Framework: BM25 and Beyond. Foundations and Trends in Information Retrieval. Explains saturation, length normalisation and the probabilistic roots of BM25.
• Salton and Buckley (1988). Term-weighting approaches in automatic text retrieval. Information Processing & Management. Provides the term-weighting background for the TF–IDF comparison.
• Luan et al. (2021). Sparse, Dense, and Attentional Representations for Text Retrieval. TACL. Extends the lexical baseline by examining sparse and dense representation limits.

Related chapters

• Similarity Is a Decision. Examines what a similarity rule preserves and discards.