← Fine Tuning From First Principles

Statistical Confidence and Ablation

Put a paired, family-clustered bootstrap interval around a difference between systems, show the same 16/24 effect producing a zero-width interval at one data seed and a wide one at another, and keep data-seed, bootstrap-seed and training-seed variation apart.

Chapter 37 ended with a difference that is arithmetically certain. On 24 test cases, the combined system succeeded on all 24 and the base system on 8. The difference is 16/24 and no further computation will change it.

The question this chapter asks is different. How much confidence should we place in an observed difference between two systems? The arithmetic is exact, but the conclusion we want to draw is usually about cases we have not seen. A result can be computed correctly and still support no broad statistical conclusion at all. The gap between those two things is set by the sampling design, the dependence between cases and the number of comparisons made. It also depends on what else changed between the two systems.

The chapter’s most useful result is uncomfortable. The same 16/24 effect, with the same 24 per-case differences, gives a 95% bootstrap interval of zero width at one data seed and an interval running from 10/24 to 20/24 at the other. Nothing about the systems differs between the seeds. Only the grouping of cases into families does.

The integrated records are FT-P7-INTEGRATED-01-38-s37 and FT-P7-INTEGRATED-01-38-s38. They compare the sealed evaluations from Chapter 37’s records at the same data seed, and the comparisons were checked to bind to those evaluation digests. As before, the adapters are constructed, not trained.

Paired differences

Two systems are evaluated on the same n cases. For case i, let sᵢ(L) and sᵢ(R) be 1 if the left or right system succeeds and 0 if not. The paired difference is

$$ d_i = s_i(R) - s_i(L) \in \{-1, 0, 1\}, $$
and the **effect** is its mean,
$$ \hat\Delta = \frac{1}{n}\sum_{i=1}^{n} d_i = \hat p_R - \hat p_L . $$
This is an absolute difference in task-success rate, in the same units as the rates themselves. It is the effect size this chapter reports. A ratio, an odds ratio or a standardised difference would answer different questions. For a decision about deploying a system, the absolute change in the fraction of tasks completed correctly is usually the quantity that matters.

Pairing matters for the variance, not the point estimate. The unpaired and paired estimators give the same Δ̂, but

$$ \operatorname{Var}(\hat p_R - \hat p_L) = \operatorname{Var}(\hat p_R) + \operatorname{Var}(\hat p_L) - 2\operatorname{Cov}(\hat p_R, \hat p_L). $$
When both systems find the same cases easy and the same cases hard, the covariance is positive. An analysis that treats the two evaluations as independent drops it and overstates the uncertainty. Pairing uses it. That is only legitimate if the cases really are the same: same inputs, same order, same labels, same contract. The implementation refuses to compare evaluations whose dataset identities differ (`PAIRED_DATA`), whose case IDs do not match (`PAIRED_IDS`), whose families or labels disagree (`PAIRED_LABEL`), or whose contract or label authority differs (`PAIRED_CONTEXT`).

The bootstrap

The bootstrap (Efron, 1979) estimates the sampling variability of a statistic by treating the observed sample as a stand-in for the population. It draws B new samples of the same size, with replacement, from the observed one. It recomputes the statistic on each, and studies the spread of the B values. A percentile interval at level 1 − α takes the α/2 and 1 − α/2 quantiles of that spread.

The implementation uses B = 400 replications and α = 0.05. Quantiles use linear interpolation between order statistics. For sorted values v₀ ≤ … ≤ v_{B−1}, the q-quantile is

$$ Q(q) = v_{\lfloor h\rfloor} + (h - \lfloor h\rfloor)\,\big(v_{\lfloor h\rfloor+1} - v_{\lfloor h\rfloor}\big), \qquad h = (B-1)\,q . $$
For the values (0, 2, 4, 6), Q(0.25) = 1.5.

Paired bootstrap resampling is standard practice for comparing systems on a shared test set. Koehn (2004) is an early and widely used statement of it for machine translation. The resampled object is the pair, never the two systems separately, so each replicate preserves the covariance.

Two cautions belong here, before any numbers. First, a percentile interval is a procedure. If the sampling assumptions held, intervals constructed this way would contain the true difference in roughly 95% of repeated experiments. A single realised interval does not contain the true value “with 95% probability”. The value is fixed, and the interval either contains it or does not. Second, the repeated-sampling guarantee is asymptotic and weak in small samples. With 8 independent units, the experiment below cannot validate the nominal coverage, and it does not try to.

The statistical unit

The bootstrap’s central assumption is that the resampled units are independent draws. The 24 test cases are not independent. They come in families of three consecutive cases. In a real dataset, a family might be several items written by the same annotator, several paraphrases of one source document, or several turns of one conversation. Cases within a family share causes of success and failure.

When observations are clustered, the variance of a mean depends on the correlation inside clusters. For m observations per cluster, with common variance σ² and intra-cluster correlation ρ, the variance of the overall mean of n observations is

$$ \operatorname{Var}(\bar d) = \frac{\sigma^2}{n}\,\big(1 + (m-1)\rho\big). $$
The factor 1 + (m − 1)ρ is the **design effect**. If ρ > 0, treating the n cases as independent understates the variance, and the interval comes out too narrow. If ρ < 0, the reverse happens. Resampling whole clusters lets the data determine the effect without estimating ρ (Field and Welsh, 2007).

The implementation therefore resamples families, not cases. With 8 families, each bootstrap replicate draws 8 families with replacement, pools their per-case differences, and takes the example-weighted mean. Because every family here has three cases, that equals the mean of the 8 family means. The random generator is Python’s random.Random, seeded explicitly, and families are drawn from a sorted key list so that the draw sequence is reproducible. A request for case-level resampling is refused (CLUSTER_REQUIRED) whenever any family contains more than one case.

What the comparisons produced

Four comparisons were declared before the run. Each was evaluated at two data seeds and two bootstrap seeds, with 400 replications each time.

  • Positive: base → combined.
  • Small: single fault → combined. The two differ in one delivered output.
  • Null: combined → combined.
  • A ablation: base → adapter A.
Comparison Effect Data seed Bootstrap seed 37 Bootstrap seed 38
Positive 16/24 37 [16/24, 16/24] [16/24, 16/24]
Positive 16/24 38 [10/24, 20/24] [10/24, 22/24]
Small 1/24 37 [0, 0.125] [0, 0.125]
Small 1/24 38 [0, 0.125] [0, 0.125]
Null 0 37 [0, 0] [0, 0]
Null 0 38 [0, 0] [0, 0]
A ablation 4/24 37 [1/24, 6/24] [1/24, 7/24]
A ablation 4/24 38 [1/24, 6/24] [1/24, 7/24]

The intervals are shown as exact fractions of 24 where they are exact. In decimals, 16/24 is 0.667, 10/24 is 0.417, 20/24 is 0.833 and 22/24 is 0.917. The authoring cross-check reimplemented the resampling, quantiles and pairing without importing the library. It reproduced all 400 draws of every comparison exactly, as well as each effect and interval endpoint.

Paired family-cluster bootstrap at two data seeds

The figure shows why the grouping decides the interval. It is derived from the recorded paired differences: every seed-37 family sums to 2, so resampling cannot vary, while seed-38 families alternate between 3 and 1, so the same 16/24 effect yields [10/24, 20/24]. The zero-width case is flagged degenerate, not precise.

A zero-width interval

At data seed 37 the positive comparison is degenerate: every one of the 400 replicates equals 16/24. The reason is visible in the family sums of the differences. The seed-37 test partition starts at pattern 1, so its families contain patterns (1, 2, 3) and (4, 5, 0) in alternation. The base system fails patterns 2 to 5 and the combined system fails none, so every family contains exactly two differences of 1 and one of 0. Every family mean is 2/3. Any resample of 8 families therefore has mean 2/3.

At data seed 38 the partition starts at pattern 2, and the families are (2, 3, 4) and (5, 0, 1). The first kind has three differences of 1, the second only one. The per-case differences are the same multiset as before, 16 ones and 8 zeros, but the family sums now alternate between 3 and 1. If m of the 8 resampled families are of the first kind, the replicate mean is

$$ \frac{3m + (8-m)}{24} = \frac{8 + 2m}{24}, \qquad m \sim \mathrm{Binomial}(8, \tfrac12). $$
The interval endpoints follow. P(m ≤ 0) = 1/256 and P(m ≤ 1) = 9/256 ≈ 0.035, so the 2.5% quantile lands at m = 1, which gives 10/24. At the top, P(m ≥ 7) = 9/256 as well, so the 97.5% quantile sits near the boundary between m = 6 (20/24) and m = 7 (22/24). Which one appears depends on the particular 400 draws. Bootstrap seed 37 produced 20/24 and bootstrap seed 38 produced 22/24.

The analytical standard errors make the contrast explicit. They are derived from the recorded differences, not from any additional run. If the 24 cases were resampled independently, the standard error of the mean of 16 ones and 8 zeros would be √((2/3)(1/3)/24) ≈ 0.096 at both seeds. Family resampling gives 0 at seed 37. At seed 38, where the family means are 1 and 1/3, it gives (1/3)/√8 ≈ 0.118. The seed-37 arrangement behaves like negative intra-family correlation, because each family balances successes against failures. The seed-38 arrangement behaves like positive correlation. The same 24 numbers carry different information depending on how they are grouped.

So a zero-width interval does not mean zero uncertainty. It means that the resampling procedure, applied to this sample, produced no variation. Real uncertainty, about other families, other patterns or other tasks, is not represented in the procedure at all. With 8 constructed families whose structure is periodic, the procedure has very little to work with. The implementation flags every such case as degenerate_bootstrap = true, so the zero width cannot be mistaken for precision.

The small control

The single fault differs from the combined system in one delivered output, so one family has a difference sum of 1 and the other seven have 0. A replicate mean is k/24, where k counts how often that family is drawn, with k ~ Binomial(8, 1/8). Because P(k = 0) = (7/8)⁸ ≈ 0.344, which is well above 0.025, the lower endpoint is 0. Because P(k ≥ 3) ≈ 0.067 but P(k ≥ 4) ≈ 0.011, the upper endpoint is 3/24 = 0.125. The interval [0, 0.125] was reproduced at both data seeds and both bootstrap seeds.

The interval includes zero, and the effect, 1/24 ≈ 0.042, is below the practical threshold of 0.1 declared in the configuration before the run. Neither fact says the fault does not matter. At seed 38 the fault executes a disallowed erase. Statistical size and practical consequence are different axes. A single unsafe action can matter more than a large average gain.

The ablation

Adapter A alone against the base differs on the 4 certain-erase cases. Those fall one to a family in alternate families at both seeds, so the family sums alternate between 1 and 0. The replicate mean is m/24 with m ~ Binomial(8, 1/2), and the interval is [1/24, 6/24] at bootstrap seed 37 and [1/24, 7/24] at bootstrap seed 38. The effect, 4/24 ≈ 0.167, exceeds the practical threshold.

Three kinds of seed

This experiment involves three different random seeds. Only two of them did anything.

  • Data seed (37 or 38). It decides how the fixture’s patterns are arranged into families. Changing it changed the positive interval from zero width to [10/24, 20/24]. Data-seed variation is variation in which sample was drawn.
  • Bootstrap seed (37 or 38). It decides which families the resampler picks. Changing it moved one endpoint from 20/24 to 22/24 and another from 6/24 to 7/24. Bootstrap-seed variation is Monte Carlo error in computing the interval. More replications shrink it, and it says nothing about the systems.
  • Training seed. It does not exist here. Nothing was trained, so there is no variation from initialisation, data order or optimisation noise. In a real fine-tuning comparison, training-seed variance is often the largest source of disagreement between runs, and a bootstrap over test cases does not capture it at all. Estimating it requires training several times.

A report that says “the result was stable across seeds” without saying which seed was varied has said very little. The limitations recorded with the experiment make the same point: the constructed models have no training-seed variability, and bootstrap variability is a different quantity.

Many comparisons

Every additional comparison is another chance for a spurious difference. If k independent tests are each run at level α, the chance of at least one false rejection when all nulls are true is 1 − (1 − α)ᵏ. The Holm procedure (Holm, 1979) controls that family-wise error rate in the strong sense, with more power than plain Bonferroni. Sort the k p-values in increasing order, p₍₁₎ ≤ … ≤ p₍ₖ₎, and set

$$ \tilde p_{(j)} = \max_{i \le j}\ \min\!\big(1,\ (k - i + 1)\,p_{(i)}\big). $$
A hypothesis is rejected at level α if its adjusted value is at most α.

The implementation checks the arithmetic on a declared fixture of three p-values, (0.01, 0.04, 0.03). Sorted, they are 0.01, 0.03 and 0.04. The multiplied values are 3 × 0.01 = 0.03, 2 × 0.03 = 0.06 and 1 × 0.04 = 0.04. The running maximum gives 0.03, 0.06 and 0.06. In the original order, the adjusted values are (0.03, 0.06, 0.06), which matches R’s p.adjust with method = "holm". The record labels these values MATHEMATICAL_FIXTURE_NOT_EXPERIMENTAL_PVALUES. The experiment computed no p-value for any system comparison, and none should be inferred.

Ablation and confounding

An ablation removes or changes one component and measures the consequence. It supports a causal reading only if exactly one thing changed. Compare two configurations:

Configuration Weight on A Weight on B
Combined 1.0 1.0
B removed 1.0 0.0
Weak B 1.0 0.3

Combined against “B removed” changes one declared variable, the B weight. The implementation records it as ONE_VARIABLE_ABLATION, and the difference in task success can be attributed to B, given everything else that was held fixed. Suppose instead that a comparison changes the B weight and the policy revision, or the B weight and the dataset. Then the implementation raises ABLATION_CONFOUNDED and refuses to label the comparison an ablation. The record keeps that diagnostic as a negative control.

Confounding also enters through the evaluation. If the left and right evaluations use different test cases, a difference in scores mixes the effect of the system with the effect of the data. That is why pairing is enforced, and why an “unpaired” comparison of two separately reported accuracies is not an ablation even when only one model component changed.

Interpreting the intervals

What may be said about the positive comparison? The combined system succeeded on 16 more of these 24 cases than the base. Under family resampling of this fixture, the interval was degenerate at one data seed and [10/24, 20/24] or [10/24, 22/24] at the other, depending on the bootstrap seed. The procedure has 8 units, a periodic structure and no training variability. Its nominal 95% coverage has not been validated.

What may not be said: that the combined system is significantly better in any population sense; that the true improvement lies in [10/24, 20/24] with 95% probability; or that adapter composition improves fine-tuned models. The experiment can support none of these. The perfect rule baseline from Chapter 37 also scored 24/24, and a comparison against it would show no improvement at all.

Practical lessons

  • Fix the statistical unit before computing anything, and resample at the level where independence is plausible. That level is usually the source document, conversation, user or template, not the individual example.
  • Pair comparisons on identical cases, and make the pairing checkable through dataset and label identities.
  • Flag degenerate intervals instead of reporting them as precision.
  • Report which seed varied. Use enough bootstrap replications that Monte Carlo error is small compared with the interval, and use multiple training runs to estimate training variability.
  • Declare the comparisons and the practical threshold before the run, and adjust for multiplicity when more than one comparison supports a claim.
  • Treat an ablation as causal only when exactly one variable changed.

Limitations

The test set has 8 families of 3 constructed cases per data seed. The adapters are constructed, so there is no training-seed variation and no learning to estimate. The percentile bootstrap with 8 clusters has unknown and probably poor coverage, and the experiment does not establish population coverage. The Holm values are an arithmetic fixture, not experimental p-values. The practical threshold of 0.1 is a declared configuration value, not a derived quantity. The records carry AUTHENTICATED_EXECUTION = NOT_ESTABLISHED.

FineTuneLab

ftlab.delivery.statistics provides compare, which checks sealing and pairing and runs the family-cluster percentile bootstrap. It also returns the per-case differences, all draws, the effect, the interval and a degeneracy flag. The module also provides holm and ablation. The experiment is ch38 in ftlab.delivery.experiments. It consumes the Chapter 37 evaluations for the same data seed.

Where this leads

Every interval in this chapter describes variation within one fixed distribution: the same six patterns, the same policy and the same request format. A well-measured in-distribution result can still fail as soon as the inputs, the policy or the surrounding system change. Chapter 39, Generalisation, Shift and Adversarial Tests, challenges the same systems with transformed inputs, an unknown operation, ambiguous requests, injected instructions and a policy revision.

Further Reading

  1. Bradley Efron, “Bootstrap Methods: Another Look at the Jackknife” (1979). Read the source. The original bootstrap paper, clarifying what a resampling interval is as a procedure with repeated-sampling behaviour.
  2. Christopher A. Field and Alan H. Welsh, “Bootstrapping Clustered Data” (2007). Read the source. The theory behind resampling families rather than cases, explaining the chapter’s zero-width versus wide interval contrast.
  3. Rotem Dror et al., “The Hitchhiker’s Guide to Testing Statistical Significance in Natural Language Processing” (2018). Read the source. A practical protocol for choosing paired tests and handling multiple comparisons in NLP settings.