← Fine Tuning From First Principles

Deduplication, Splits and Contamination

Protect evaluation with lineage-aware grouping, distinguish deliberate policy pairs from leakage, and test attacks on split authority.

A model answers every test incident correctly. Then we discover that each test example is a capitalised copy of an incident in training. The score was computed correctly; its interpretation was wrong. It measured recognition of previously available content while being presented as performance on unseen cases.

Leakage can survive valid schemas, distinct record IDs and different file hashes. It can also occur without a single gradient update: a held-out example may enter a retrieval index, a few-shot prompt, a label lookup or repeated hyperparameter selection. Dataset separation is therefore a property of relationships and permitted uses, rather than a property of directory names.

The running pipeline now has provenance, labels, formats and exact supervision. We will attack the remaining boundary: which examples may inform development, and which observations can support a subsequent evaluation claim?

The 48-record allocation and lookup measurements below are preserved pilot results. The final section replaces its invalid role design with revision 0.2; the pilot is restricted to mechanism demonstrations.

Start with the question a split is meant to answer

A random row split can estimate performance on new rows from a similar mixture under some assumptions. It cannot automatically estimate performance on new people, new scenario families, future time periods or unseen policies. Each target population implies a different grouping and sampling problem.

The original pilot test separated declared authored families from development families; this alone did not establish a valid evaluation design. A family challenge and retention proposal reserve additional groups. The policy-shift proposal contains explicit p1/p2 partners inside one protected group. These names describe intended purposes; the tiny synthetic groups do not establish realistic out-of-distribution performance.

A temporal split can address a future-use question, but only if timestamps represent the relevant availability boundary. Later annotation or retrospective metadata can leak future information into earlier inputs. A policy-version split tests responsiveness to changed policy only when that policy is actually supplied and the stable incident evidence is controlled.

The broader methodological problem is documented in Kapoor and Narayanan’s leakage study. Its relevance here is the need to inspect how information became available, not to infer that our small audit eliminates every category of leakage.

Several kinds of equality matter

A file-byte hash identifies exact bytes. A canonical-object hash can ignore chosen serialisation differences. A text normaliser may ignore case, whitespace and punctuation. A semantic incident ID identifies a declared incident independent of active policy. A family ID groups variants descended from one scenario construction.

None replaces the others. An incident ID can change while the text stays equal. Two paraphrases can differ after every lexical normalisation. Conversely, deleting punctuation can conflate legitimately different inputs. Our normaliser uses Unicode NFC, case folding and word-token extraction. It catches the intended case/whitespace/punctuation variants but does not prove semantic equivalence.

Identity Answers this question Misses this question
File-byte hash Which exact bytes were ingested? Reindented copies or the same text under a new ID
Canonical-object hash Which declared normalised object was interpreted? Two paraphrases with equal meaning
Text normaliser (NFC, case, words) Which declared case/whitespace/punctuation variants match? Synonyms, translation and short negations
Semantic incident ID Which declared incident is this, independent of policy? Whether two reports describe one real incident
Family ID Which declared construction lineage produced this? Undeclared common ancestry outside the captured graph

For word sets $A$ and $B$, the implemented near-duplicate candidate score is

$$ J(A,B)=\frac{|A\cap B|}{|A\cup B|}. $$
A threshold of 0.85 flags review candidates. It is a declared engineering heuristic, not a measured optimal threshold or proof that lower-scoring examples are unrelated. Synonyms, translation and short negations can defeat lexical similarity. MinHash can approximate set similarity efficiently for large corpora, but approximation does not turn lexical overlap into a semantic oracle. Our small corpus uses exact pairwise calculation rather than an unnecessary approximate index.

For $n$ records, pairwise comparison costs $n(n-1)/2$ comparisons. That is manageable for 48 records, or1,128 pairs. Large datasets need blocking or indexing plus a review strategy, with recall limitations stated. Optimising that search before defining the relation being sought would make an unclear audit faster rather than more trustworthy.

Split connected groups before trusting the rows

Construct a graph whose vertices are records. Connect records sharing family, template, entity group or canonical semantic identity, and connect each transformation to its parent. A connected component is an indivisible assignment unit. Transitivity matters: recordA can share a template withB, which shares an entity withC, even whenA andC do not share a field directly.

The pilot grouping implementation uses union-find to obtain components. It then sorts components by a seed-bound hash of their family identities and allocates them cyclically to six proposed roles. This is deterministic allocation, not a statistical guarantee or a promise that every class appears in every split. Group-based cross-validation documentation provides the general motivation for keeping related samples together; our allocator is a small custom implementation and does not call scikit-learn.

Three surface variants of one authored incident therefore stay together. Changing a name does not erase a retained parent link or shared template. Missing parents are rejected instead of silently treating a derived row as independent. QA also checks the generator’s declared family authority, so a submitted family rename cannot quietly rewrite this candidate’s construction history.

We retain duplicate records and their lineage rather than silently dropping them. A normalised duplicate may be an intentional robustness probe. Its presence affects weighting and effective sample size, so it must be counted and grouped. Deduplication policy can later remove or weight it in a new dataset revision; it should not alter the held-out set invisibly.

Watch a useless memoriser earn a perfect score

The negative demonstration makes an intentionally invalid split. For each of 14 authored p1 families, it puts the original and paraphrase in a lookup’s training data, then evaluates the capitalised duplicate. A dictionary keyed by normalised incident text retrieves the stored answer.

The executed lookup returns14 correct answers out of 14. No model was trained and no language understanding was required. Case folding alone exposed the answer. This is exactly why a headline number must include a statement about what information was previously available.

Under the actual family-grouped proposal, the same kind of lookup gets0 of 6 ordinary test examples: those families’ text keys are absent from the proposed training subset. The two demonstrations use different evaluation sets, so their difference is not a matched estimate of an ML algorithm’s quality. It is a controlled illustration of how the evaluation question changes when related content crosses a boundary.

Group first, then allocate; block every leakage path

The left panel groups related records into one indivisible component before allocation; the right panel blocks the four leakage paths that need no optimiser. Notice the pilot contrast the figure records: 14 of 14 retrieved once leaked, 0 of 6 once grouped — an illustration of the question change, not a model comparison.

Neither demonstration is an authorised production benchmark. All examples are public synthetic teaching fixtures undergoing candidate QA. Deliberately constructing a leaked fixture here tests the audit; it does not licence later training code to read protected data.

Treat a policy pair as a declared relationship

The same incident under p1 and p2 is supposed to share text and stable labels. A naive exact-duplicate rule would flag the pair without understanding its purpose. Ignoring every pair with different policies would make the opposite error: it would let a training copy of a protected incident hide behind a new policy version.

We recognise a declared policy pair only when the two records belong to the same family, occupy the same protected policy_shift proposal and carry different policy revisions. Its route and citation must still validate under the appropriate derived task contract. Moving one partner into training produces group and exact-overlap failures. Moving the whole paired family into training produces an undeclared-policy-pair failure. Renaming the split does not cure either problem.

For the measured candidate, the policy-shift proposal contains two families, each with three p1 surfaces and three p2 partners. One is the access incident whose route changes; the other is a contradictory incident that continues to require human review while its citation changes. Six added p2 records make the candidate total48.

This tests stable-versus-policy-dependent semantics. It is not evidence that those six policy-shift pairs support a reliable estimate of adaptation performance. Their dependency is part of the design and would need to be respected by later statistical analysis.

An alias cannot grant permission

Chapter06’s canonical split-manifest resolver and access ledger remain the authority for data use. The new grouping code produces proposals. It does not replace that authority with a second boolean called safe_for_training.

The production task contract is still draft and contains no approved split identities. Exported provisional manifests therefore fail the existing resolver’s non-development freeze requirement. The capstone requests access through that interface for all six roles, including an innocuous-sounding alias; all are denied. An alias is a name for an artifact, not a way to change its role or create task-owned approval.

Candidate QA needs to inspect inputs and labels to find defects. We record that purpose explicitly. Those inspections mean these small examples are available development fixtures, not a secret untouched benchmark waiting to be reused as if nobody had seen it. A future study needs an approved access boundary and a defensible evaluation basis.

Our QA exposure audit additionally rejects training, tuning, prompt examples, retrieval and preference generation involving protected proposals. These checks show that leakage need not involve the optimiser. An evaluation set repeatedly consulted to choose hyperparameters has informed the choice even if its rows never enter a minibatch.

Attack the whole chain

The targeted probes change IDs, case and punctuation; create high-overlap new records; move family siblings; conceal gold lookup behind aliases; misdeclare policy pairs; change tokenizer or supervision identities; and mutate sealed labels, splits and cards. Each probe records its input change, expected diagnostic and actual outcome. A rejection counts as success only if it occurs for the intended reason.

The candidate itself contains26 normalised duplicate/policy-pair relationships retained for explicit review and no additional near candidates at the declared threshold. Zero lexical candidates is not proof of semantic independence. Separately constructed attacks demonstrate that high-overlap foreign records are flagged, while known paraphrases remain grouped through declared lineage.

A remaining risk is undeclared common ancestry outside the captured source graph. No hash can establish that two unknown natural-language reports describe different incidents. That requires trustworthy collection metadata, semantic investigation and appropriate review. The next chapter assembles the checks into a release process that keeps this uncertainty visible, rather than turning a green schema check into authority to freeze a benchmark.

Repair the question before repairing the allocation

The 48-record pilot assigned hash-sorted connected components round-robin. Its nine training records contained six service and three unknown examples: access, security and billing appeared only outside training. A perfect family-boundary check concealed an unsuitable supervised-learning comparison. Those historical lookup scores and pair counts above describe the pilot, not the replacement candidate.

Revision 0.2 fixes the assignment plan before generation. For each of five classes it declares seven distinct mechanism families: two for training and one each for validation, ordinary test, family challenge, policy shift and future retention probes. Each family contributes two related surfaces. Policy-shift families additionally contribute two p2 partners. Infeasible class coverage fails SPLIT_ALLOCATION_INFEASIBLE; moving rows later cannot silently relax the design.

Role Records per class Families per class Intended question
Train 4 2 Available for future parameter updates after authorisation
Validation 2 1 Future development and model selection
Ordinary test 2 1 Distinct declared mechanisms within the same taxonomy
Family challenge 2 1 Declared compositions of the two training mechanisms
Policy shift 4 1 Two controlled incident surfaces under p1 and p2
Retention probes 2 1 Future capability probes; no retention measured

For example, access training covers credential expiry and inherited denial; its challenge composes those mechanisms. Renaming a training template would not create a new mechanism. The design records mechanism, temporal state, entity type, impact, missing-information condition and composition. Equality of complete fact signatures across families or roles raises SEMANTIC_OVERLAP_NEEDS_REVIEW. Coarser related signatures are also reported without treating related cases as duplicates. These checks are conservative review aids, not a natural-language equivalence oracle.

Class coverage asks whether the learning problem is represented. Group separation asks whether declared dependencies cross a boundary. Semantic novelty asks what differs in the underlying situation. Sample-size adequacy asks whether an estimate can answer a specified scientific question with useful uncertainty. None implies the others. Two surfaces of a mechanism are not two independent mechanisms, and ten policy pairs are not twenty independent incidents.

All replacement examples have been inspected for construction and QA. No untouched final benchmark is claimed. Protected-role exposure to training, prompt selection, retrieval or preference generation is rejected. A future ordinary test needs an approved access boundary and sampling argument; repeated tuning on these public fixtures would make them development data. Retention remains FUTURE_RETENTION_PROBES_NOT_MEASURED until an earlier capability and comparison protocol exist. A separate policy-free parity demonstration has train/validation/test roles only; it fabricates no policy-shift task.

Further Reading

  1. Katherine Lee et al., “Deduplicating Training Data Makes Language Models Better” (2021). Read the source. Measures duplicates and train-test overlap and shows deduplication reduces memorisation, supporting lineage-aware grouping.
  2. Sayash Kapoor and Arvind Narayanan, “Leakage and the Reproducibility Crisis in ML-based Science” (2022). Read the source. Catalogues eight leakage types across hundreds of papers, justifying exposure audits that go beyond the optimiser.
  3. Scikit-learn contributors, “Cross-validation: evaluating estimator performance” (living documentation). Read the source. Explains group-wise and time-series splitters, motivating indivisible assignment units; the book uses its own union-find allocator rather than calling this library.