Controlled Experiments and Regression Gates
Freeze a retrieval comparison, isolate observation-conditioned stopping and retain query-level gains, losses and resource differences.
The controller in Bounded Adaptive Retrieval: When to Stop recovered a relevant GIMP document after its first lexical search missed. That is a real success in the recorded trace. It is also an attractive story to tell about adaptive retrieval—especially if we stop examining the evidence at that one query.
The fixed final comparison changes the interpretation. Across the full question set, stopping early saves retrieval calls but loses more judged ranking quality than it gains. The GIMP case explains how continuation can help; it cannot establish that this particular stopping policy is better overall.
Metrics that Answer Different Questions defined what each score measures. We now need an experiment that distinguishes the effect of stopping from the effects of retrieval method, candidate depth and selection budget. Otherwise a difference in the final metric may have several competing explanations.
Make the proposed explanation testable
The hypothesis is deliberately narrow: an observation-conditioned stopping rule may avoid some dense searches without giving up too much useful evidence. That is a proposed quality–cost trade-off, not a claim that adaptation must beat every fixed retrieval route. A fair test must measure both what the rule saves and what its skipped searches would have recovered.
Before comparing methods, we freeze the roles of the data, available tools, judgement provenance, budgets, baseline policies, selection procedure, metrics, stopping rules and cost methodology. The implementation and development checks receive a separate later seal. After choosing a threshold on the designated policy-selection questions, we seal that choice before running the final comparison.
Each seal answers a different audit question. The protocol seal records what comparison was planned; the implementation seal identifies the code subjected to that plan; the selection seal records the threshold chosen without reference to final scores. None creates independent judgements, a representative deployment sample or evidence that the model never encountered related training material. Those limits remain as described in Relevance Judgments and Test Collections.
The two declared stopping thresholds, .5 and .8, were compared on eight policy-selection questions. Threshold .8 achieved the higher mean nDCG on that slice and was selected by the predeclared rule. Its later failures are part of the result: we do not add a tango-specific exception or search for a more favourable threshold after viewing the final questions.
Compare plausible baselines
The main final comparison holds the 34 questions, trusted external view, document judgements and whole-unit selection cap fixed. Every policy is subject to a maximum of two retrieval calls, six returned candidate rows and four selected units within 4,000 characters. These common limits define the allowance, not an assertion that every policy uses the same computation.
| Policy | Retrieval procedure |
|---|---|
| Fixed lexical | One lexical search, depth six |
| Fixed dense | One exact dense search, depth six |
| Fixed hybrid | Lexical three plus dense three, then RRF |
| Existing router | The earlier deterministic surface rules |
| Adaptive | Lexical three, then observed stop or dense three |
| Always-continue ablation | Same adaptive text procedure with its stop test disabled |
A fixed lexical or dense search may spend all six candidate rows on one representation. Adaptive continuation instead takes up to three lexical rows and then, conditionally, three dense rows. A comparison between adaptive and fixed lexical search changes depth allocation and potentially the representation as well as stopping. It is useful for comparing systems, but it does not isolate the causal effect of the stop rule.
The always-continue ablation supplies the tighter control. Both procedures begin with lexical depth three; if allowed to continue, they use the same dense depth three, fusion and whole-unit packing. The relevant change is whether the observed lexical results can trigger an early stop. This pair therefore isolates the stopping decision more directly on the external text-search slice.
Book result: comparison-policy control. A separate audit of Next illustrates why an apparently reproducible comparison can still be invalid. Its passive arm checks ground_truth != "irrelevant" before applying a score threshold of 40. In the constructed control, changing only the reference label changes the recommendation. The reference judgement has leaked into the decision being evaluated, violating the separation required in Relevance Judgments and Test Collections.
The other Next arm changes more than access to evidence. It multiplies the initial score by (.5 + .5 × evidence_quality) and requires a value above 50, while its evidence-quality input is simulated from metadata rather than fetched documents. The retained constructed control starts with score 60 and quality .3, producing 39: the candidate fails before any document retrieval. This cannot isolate the benefit or harm of actual retrieved evidence.
The frozen code controls and the recall ledger in Metrics that Answer Different Questions make these implementation choices auditable. They do not establish that retrieval itself caused the observed recommendation differences. That question would require holding the candidate cohort, assessor and decision rule fixed while changing only the evidence supplied. Correct arithmetic is necessary, but it cannot rescue a confounded comparison.
Preserve the complete outcome
The frozen external result is:
| Policy | Recall@4 | MRR@4 | nDCG@4 | Mean calls |
|---|---|---|---|---|
| Lexical | .1699 | .7451 | .3152 | 1.000 |
| Dense | .2030 | .7672 | .4180 | 1.000 |
| Hybrid | .1880 | .8088 | .3872 | 2.000 |
| Existing router | .1880 | .8088 | .3872 | 2.000 |
| Adaptive | .1622 | .7500 | .2774 | 1.382 |
| Always continue | .1880 | .8088 | .3872 | 2.000 |
All reported values are means over the same 34 externally judged needs, using the declared convention that unjudged results receive zero gain in these point estimates. Recall@4, MRR@4 and nDCG@4 measure recovery and ordering of annotated documents or entities. They do not measure complete source bundles, generated answers or downstream helpfulness.
The existing router selects its hybrid fallback for every final question, so its identical results do not demonstrate effective route selection. Always continuing also produces exactly the same ordered selected references as fixed hybrid search on this slice. The three matching rows are different procedure labels applied to the same observed outcomes, not three independent confirmations of hybrid quality.
Adaptive uses 47 retrieval calls, compared with 68 for always continuing, but its mean recall and nDCG are lower. It therefore demonstrates call savings, not an overall quality gain. Fixed dense reaches mean nDCG@4 .4180 and fixed lexical .3152, both above adaptive’s .2774, with only 34 calls each. A logical call is not a common unit of elapsed time or CPU work. Dense leads these policies on nDCG; hybrid leads on MRR. Which trade-off matters depends on the declared task and operating constraint.
Find the queries behind the averages
The aggregate means hide substantial query-level variation. Dense beats lexical in nDCG on nineteen questions, loses on nine and ties on six. Hybrid records seventeen improvements, seven losses and ten ties in the retained comparison. Thus the methods genuinely differ on this sample; selecting a favourable example is not the same as establishing that a policy selects the favourable method consistently.
Against always continuing, adaptive has two nDCG gains, sixteen losses and sixteen ties across the 34 questions. It stops after the first action on 21 questions; judged retrospectively, the skipped call would have improved sixteen of those, left three unchanged and worsened two. Those labels explain the failure after the experiment. They were not available to the controller when it decided to stop.
Every early stop returns the first three lexical rows, while the main evaluation measures up to four results. That difference in prefix length could contribute to the loss. To check it, we compare both methods at a common three-result cutoff: continuation still improves twelve stopped queries, ties six and worsens three. The effect is therefore not solely the missing fourth position. Three cases counted as improvements at cutoff four become ties at cutoff three, and one becomes a loss; aggregate subtraction alone does not identify four cases harmed only by packet length.
Eighteen of the 21 stopped prefixes already contain at least one known relevant document. Yet the median stopped need has 26 known relevant documents. The overlap proxy can recognise vocabulary without knowing whether enough relevant evidence has been collected or whether higher-grade evidence remains elsewhere. The external document judgements reveal this missed opportunity retrospectively, but they still do not specify a sufficient answer bundle. A first relevant result is not proof that a multi-part need is complete.
The fixed lexical depth-six baseline also challenges the early stop: it beats adaptive on twelve of the 21 stopped questions and ties on nine at nDCG@4. Across all 34 questions, adaptive improves on lexical five times, loses fifteen times and ties fourteen times. Because lexical depth six and adaptive depth three allocate their candidate budgets differently, this is a useful practical comparison rather than a pure test of the stop condition. The always-continue pair supplies that more focused ablation.
Headroom depends on the metric. Lexical already ranks a known positive first for 23 of 34 questions, leaving eleven with room to improve that particular position. Yet no lexical nDCG value is already one, so graded ordering can still improve. Twenty-six questions have more than four known positives, making full known-positive recall impossible at cutoff four. None of these document-level facts says that all those positives are jointly required to answer the question.
Control resources without pretending they are identical
A shared maximum budget prevents any arm from exceeding the declared allowance. It does not make the arms equally expensive or equally informative. Dense and lexical calls perform different computation, two generators may return overlapping units, and a shorter selected packet can omit valuable evidence. Report both the common caps and what each method actually consumes.
Selected-text consumption differs as well: approximately 1,068 characters per query for adaptive, 1,240 for hybrid, 1,159 for dense and 1,279 for lexical. A smaller packet can be desirable, but it is not automatically a better use of resources when judged evidence is lost. Conversely, the value of a second retrieval opportunity cannot be credited solely to a better ranking rule without accounting for that extra action.
The budgets were chosen from development-unit lengths before the final outcomes were inspected. We retain those limits, the actual usage and every unfavourable result. Raising the cap or constructing a replacement test after seeing the losses would be a new experiment, not a correction to this frozen comparison.
Build gates around different promises
Regression checks protect different layers of the claim. Source identity, eligibility, exclusion of judgement fields from controller requests, bounded execution and preservation of earlier evidence require exact checks. Deterministic replay should reproduce the retained units and observations in the recorded environment. Fresh neural inference instead needs its declared numerical tolerance; reusing saved vectors cannot stand in for that execution. A passing check at one layer does not establish the next.
The diagram shows the order of checks and decisions. Passing a technical check allows the next question to be asked; it cannot supply independent review or author approval by itself.
%%{init: {"flowchart": {"wrappingWidth": 470}}}%%
flowchart TD
G1["Exact checks: source identity, permitted view, no label fields in requests, bounded loops, unchanged prior evidence"] --> G2["Deterministic replay: retained units and observations on the recorded environment"]
G2 --> G3["Fresh neural execution: its own stated numerical tolerance"]
G3 --> G4["Metric gates: declared thresholds, unknown-label policy, denominator and cost policy"]
G4 --> G5["Independent review"]
G5 --> G6["Author decision, then publication — never automatic"]
A passing metric gate confirms that a declared numerical rule was satisfied under the chosen labels, denominators and cost assumptions. It is not a verdict that the quality–cost trade-off is suitable for the reader’s application. The decision needs the query-level losses and the evidence limitations as well as the aggregate number.
Interpretation remains a separate gate. Higher MRR with lower recall, for example, means that first relevant positions and broader recovery have changed differently; neither metric may be substituted for the other. Any acceptance threshold needs a stated use case, unknown-label policy, denominator and cost accounting. Tests can detect violations of those rules, but only an explicit author decision can approve the resulting trade-off for publication.
For this experiment, the full test suite, judgement validation, independent metric arithmetic and retained-vector replay pass. The arithmetic audit checks 2,448 metric values along with the separate constructed bundle and path controls. These checks support local calculation correctness and reproducibility under the recorded conditions. They do not substitute for the independent review gate shown above.
Keep uncertainty proportional to the design
The 34 inherited public questions are not a probability sample of future deployment requests. We therefore report paired query differences and concrete failures without inferring population-level significance. More timing repetitions would improve timing description, not increase the number of independent relevance questions. Selecting favourable subgroups or metrics after seeing the final results would create another route to an overstated claim.
The supported conclusion is specific: this fixed, inspectable overlap-based stopping policy reduces retrieval calls but stops early often enough to lower judged retrieval quality on the final sample. It obeys its execution contract and still loses to simpler alternatives on key metrics. A better stopping policy remains a research question requiring a new development and evaluation cycle, not an improvement established by calling this controller adaptive.
Latency, Cost, and Operational Reliability examines whether the saved logical calls translate into meaningful time or operational savings. Only then can we discuss whether the observed retrieval-quality loss is an acceptable trade-off under a particular constraint.
References and further reading
Background and extensions for this chapter:
- Sandve et al. (2013). Ten Simple Rules for Reproducible Computational Research. PLOS Computational Biology. Supports recording the exact workflow, inputs, software and intermediate results.
- Reimers and Gurevych (2017). Reporting Score Distributions Makes a Difference: Performance Study of LSTM-networks for Sequence Tagging. EMNLP. Extends the experimental design to repeated runs when a method is stochastic.
- Lin et al. (2021). Pyserini: An Easy-to-Use Python Toolkit to Support Replicable IR Research with Sparse and Dense Representations. arXiv preprint, v1. Provides an IR-specific example of shared baselines and automated reproducibility checks.
Related chapters
- Change One Thing. Develops controlled changes and the interpretation of experimental arms.