← Retrieval From First Principles

Rerankers and Cross-Encoders

Measure a pinned local cross-encoder while preserving the candidate ceiling, input limits and actual cost.

For SciFact claim 1368, the reranker receives fifteen candidate documents, but the one annotated positive document is not among them. The model may change their order, but it cannot place an absent document in the results. The frozen run makes that limit visible: the order changes, yet judged-positive recall remains zero. This is a failure of candidate availability before it is a test of reranking quality.

Learned Sparse and Multi-Vector Retrieval considered representations that could change which documents enter the candidate pool. Here we hold that pool fixed. The question is whether paying for a joint query–candidate comparison moves existing evidence into better positions—and whether any gain is large enough to justify the additional work.

Spend interaction after retrieval

The dense retriever encodes each document without seeing the current query. Its document vectors can be reused across requests; the query vector is compared with each candidate by the declared cosine-scoring rule. This separates representation from the query-time comparison and avoids running a joint language model over every query–document pair in the corpus.

A cross-encoder receives the query and one candidate together. It can model interactions between their words before assigning a score, rather than comparing two independently produced vectors. That extra interaction costs a model evaluation for each supplied pair. In this pipeline the cross-encoder orders candidates already retrieved; it neither searches the corpus nor creates evidence missing from its input pool.

The executed model is the locally cached cross-encoder/ms-marco-MiniLM-L-6-v2. We verified its canonical pinned model revision before running inference. The model card declares MS MARCO training and an Apache-2.0 licence; its published benchmark numbers do not describe this experiment. Our measurements come from running the six-layer, single-logit cross-encoder on the whole-document candidates defined in Chapter 11.

Make the pair and its score reproducible

The input is precisely (query text, exact unit text). For SciFact, the unit text is its declared title-and-abstract representation—not a generated summary or a query-specific rewrite. The pinned tokeniser applies longest-first truncation to a maximum of 512 wordpieces across the pair. The model then produces one raw logit through an identity activation. This specification matters because a different pair representation or truncation rule would be a different scoring experiment.

Each stored score is tied to a model fingerprint, a query-text digest and a unit-text digest. The reranking function verifies all three before using it. Matching passage identifiers do not make scores reusable when the question, model or source text has changed. If a score is missing, the function fails explicitly instead of quietly mixing scoring conventions or falling back to a different one.

The reranker copies the incoming candidates, attaches the logits and sorts by descending score, breaking ties with the complete source reference. Its output must contain exactly the same reference set as the union it received; tests and every retained query check this invariant. The original BM25 and dense ranks and scores remain attached so a changed position can be explained. None of this ranking logic confers permission: the current-view projection still checks eligibility before evidence is exposed.

The logit supplies a relative ordering within this declared model and query. Its sign is not a tested threshold for admitting or refusing evidence, and a positive value is not a calibrated probability that the passage is sufficient or true. We fitted no acceptance threshold. Reading such a decision into the score would claim more than this experiment measures.

Nogueira and colleagues’ sequence-to-sequence ranker illustrates a different pair-scoring interface. It trains T5 to predict the single-token relevance labels true and false, then normalises just those two target logits. For logits a and b, the reported score is exp(a)/(exp(a)+exp(b)), or equivalently 1/(1+exp(b-a)); the ordering depends on their difference. Here true means the model’s trained relevance label, not verified factual truth or a completed answer. This is published background, not our executed single-logit model. Both methods jointly process a query–document pair, whereas ColBERT scores separately encoded token representations through late interaction. Substituting either approach would require its own pinned model and evaluation.

Give the model room, then inspect the outcome

At each fixed generator depth, the cross-encoder scores every document in the corresponding exact-reference union and then reorders that same pool. The external comparison below distinguishes the pool’s recall ceiling from recall achieved by its first three ranked documents:

Depth per generator Union recall ceiling RRF Recall@3 Cross Recall@3 Cross MRR@3 Cross nDCG@3
1 .8333 .8333 .8333 .8333 .8333
3 .8500 .8333 .8333 .8333 .8333
10 .8667 .8333 .8333 .7917 .8026

The relevance metrics average the twelve sampled SciFact claims with at least one annotated positive document; the other eight are recorded separately. These are document-level measures. Union recall describes which positives are present anywhere in the candidate pool, while Recall@3 measures which survive into the selected top three. The twenty-query sample is small, uses public labels and is not SciFact’s hidden official test. Its results should not be read as a broad domain comparison.

At depth ten, the cross-encoder changes the full candidate order on all twenty claims. Yet its three aggregate ranking metrics equal RRF’s, and its MRR@3 and nDCG@3 remain below BM25’s .8333. Reordering is observable activity, not evidence of improvement. For claim 1337, the annotated positive remains second. For claim 873, two of five positives are available in the pool but neither appears in the top three. The cross-encoder therefore leaves the specific final-selection failures visible rather than correcting them.

There is little first-relevant-rank headroom in this particular sample: BM25 already places a positive first for ten of the twelve positive claims. Only claims 1368 and 873 could improve that first-positive position, and claim 1368’s positive is missing from every tested candidate pool. At depth ten, claim 1337 alone accounts for the cross-encoder’s MRR@3 and nDCG@3 deficits relative to BM25; the other eleven positive claims have identical values for the three reported cutoff metrics. No positive claim differs on those metrics at depths one or three. This explains why the aggregate comparison is largely unchanged. It neither proves reranking generally ineffective nor limits improvements in deeper recall, nDCG or coverage when several positive documents matter.

The fictional development collection gives a contrasting result. Among nine answerable section-based needs, depth-ten RRF covers 7/9 required evidence bundles within the top three, BM25 covers 8/9 and the cross-encoder covers 9/9. The cache-purge need s-d04, for example, becomes covered when reranking promotes sufficient evidence already present in the broader pool. This is an executed success on an authored teaching fixture, not independent evidence that the model will improve retrieval on external tasks. The two results can coexist because their collections, judgements and evaluation units differ.

Account for the larger pool’s actual cost

The cost measurement uses CPU inference with one Torch thread and batches of eight. Following one neutral warm-up, each query/depth prediction call is timed once. Model loading and the separate token-count audit are excluded. The values below are measured local wall times for cross-encoder prediction—not repeated latency distributions or end-to-end service timings:

Depth per generator External pairs across 20 queries Truncated pairs Median query prediction time
1 31 7 .138 seconds
3 99 12 .526 seconds
10 334 46 1.831 seconds

Total measured prediction time at generator depths one, three and ten is 3.276, 11.499 and 36.894 seconds respectively. The script invokes prediction separately for each depth; it does not compute the largest union once and reuse its scores for smaller subsets. Thus repeated pairs are paid for again across the three measurements. These times exclude acquisition, indexing, dense retrieval, token auditing and policy projection, so they cannot be added up and presented as an end-to-end search latency.

At depth ten, forty-six of the 334 external query–document pairs exceed the joint 512-wordpiece input limit. The cross-encoder scores the truncated pair even though the evidence record still retains the complete source text. The retained token lengths and truncation flags expose this difference between available evidence and model-visible evidence. We did not isolate the effect of truncation on ranking, and a longer input window has not been tested as a remedy.

Two panels. The left plots median cross-encoder prediction time per query at generator depths one, three and ten: 0.138, 0.526 and 1.831 seconds, with 31, 99 and 334 scored pairs and 7, 12 and 46 truncated pairs annotated above each bar. The right plots the union recall ceiling rising from 0.8333 to 0.8667 while RRF and cross-encoder Recall@3 stay flat at 0.8333.
Recorded cost against recorded ordering. Prediction time grows roughly thirteen-fold between depths one and ten while the achieved top-three recall does not move; the rising line is what the candidate pool made available, not what the reranker delivered.

The retained logits and pinned file hashes document an actual inference run; executable replay checks a different property. Sixteen fresh dense spot encodings reproduce their retained components exactly. One cross-encoder pair per collection was freshly scored, with a largest absolute difference of approximately 5.96e-8 against the retained score, below the declared 1e-5 tolerance. These limited checks support the recorded execution but are neither a second complete inference run nor evidence of bit-identical results across hardware or dependency versions.

Know what ordering has left undecided

A reranker can move a candidate up or down; it cannot recover a missing document, restore text omitted from its own input, verify an authority or decide that an answer is unsupported. All eight external claims without annotated positives still return candidates, but absent positive annotations do not prove those returns are erroneous. The fictional battery request is a separate, explicitly authored no-answer case: its sufficient-evidence bundle is empty, yet candidates still appear. Ordering and deciding whether enough evidence exists remain different tasks.

Even the best available order does not determine the right set of evidence to pass onward. Three highly ranked documents might repeat one fact, omit a complementary condition or consume far more text than a downstream reader can admit. Diversity, Coverage, and Evidence Budgets keeps this largest-pool ordering fixed and tests what survives when selection has an explicit content budget. The next decision is not simply which item scores highest, but which combination of evidence is worth the available space.

References and further reading

Background and extensions for this chapter: