Synthetic Data, Quality Assurance and Dataset Cards
Assemble provenance, annotations, formats, supervision and group audits into an immutable synthetic release candidate with an explicit freeze gate.
Every row in a dataset passes its schema. Its category strings are legal, its policy citations are correctly formed and its JSON is well ordered. Yet a retrieval index contains the proposed test records. The dataset must fail its release gate even though no individual row needs repair.
That example separates record validity from study validity. A trustworthy data package needs evidence about origins, meanings, relationships, transformations and permitted use. It also needs a way to stop. Reaching the last chapter of data preparation cannot itself authorise a final benchmark freeze.
We now assemble the previous five chapters into one bounded synthetic candidate. The result is reproducible and inspectable, while its lack of human adjudication, statistical adequacy and production authority remains explicit. The task contract is not rewritten to make those missing approvals disappear.
The first measured construction below is the preserved 0.1.0 pilot. Its apparent readiness was disproved by coverage, identifier and authority attacks. The final section presents the separate 0.2.0 candidate; neither artifact is an authorised benchmark.
Plan coverage before multiplying templates
A generator can cheaply create ten thousand sentences that test one rule. Their count gives little information about the diversity of situations represented. Begin with the behaviours and failure boundaries the dataset is intended to exercise, then choose controlled variations that probe them.
The original engineering pilot plan specified 14 authored scenario families: single-user and multiple-user access problems, active service outage, service degradation, active security compromise, billing, missing information, ambiguity, negation, historical compromise, contradictory evidence, multi-intent reports, embedded hostile instructions and unfamiliar access wording. The four known categories, three known severities and required abstention outcomes appear across that plan.
Each family receives three surfaces: an original, an uppercase/spacing variant and an authored paraphrase. These are deliberately related examples, not independent observations. Forty-two p1 records result. After group assignment, the two policy-shift families receive p2 partners for all three surfaces, adding six records.
This yields48 examples because of the coverage construction, not because48 is statistically sufficient for a later superiority claim. A larger benchmark would require a separate adequacy rationale, uncertainty protocol and evaluation population. Multiplying these same templates would not supply that missing justification.
Keep the generator’s answer separate from evidence about it
The generator records version, code digest, configuration, seed, family, transformations, policy revision, canonical record and source identity. Labels are rule-generated from authored facts. The generator has no access to external customer tickets and makes no model API calls.
That construction makes labels auditable, but it also creates a circularity risk. If the generator and validator share the same mistaken rubric, every example can pass while teaching the wrong task. Our ordinary semantic validator independently checks contract shape, policy routing and abstention relations, but cannot determine the truth of arbitrary free text.
QA therefore adds a separately written table of expected category/severity pairs for the 14 authored phenomena, alongside the task-contract checks. The negation and historical examples expect access/low, rather than treating every compromise word as active security. The hostile-instruction example remains security/high and cannot select its preferred billing route. These are implementing-author reference fixtures, not independent human gold or a comprehensive natural-language oracle.
Keeping those distinctions visible is useful. A future human review can inspect the disputed meaning rather than discovering that “gold” merely meant the generator’s own return value. Manual review, automated consistency and independent adjudication are different evidence classes, with different costs and failure modes.
Trace every dependency through the package
The capstone calls the same source-eligibility, annotation, rendering, supervision, grouping and audit functions tested earlier. For every record it checks source eligibility again, validates the bound annotation under the active contract/policy, reconstructs the rendered answer, and recompiles the exact token mask.
A changed annotation invalidates its dependent rendering. A changed tokenizer or supervision policy invalidates cached token evidence. A family reassignment must match the declared seed and grouping algorithm; moving an entire intact family cannot quietly preserve a claim that the old split configuration was used. The dataset card must describe the same revision and digests as the audited candidate.
The production access request remains separate. Provisional manifests cannot grant training or evaluation permission under the draft contract. The candidate exposure ledger records QA inspection of all 48 examples. These are intentionally inspectable teaching data, not protected secrets whose independence can be recovered by putting them in a directory called test afterwards.
The pipeline performs no incident-model training. Its full rendered incidents exceed the earlier educational model’s32-position capacity. The small CPU gradient comparison verifies supervision machinery on a short conversation, while the incident masks are checked directly. This boundary prevents a format integration claim from becoming an unsupported training-compatibility claim.
Read the measured composition
The preserved engineering pilot, revision 0.1.0, produced the following historical measurements:
| Quantity | Observed value |
|---|---|
| Records | 48 |
| Scenario families | 14 |
| Access / unknown / service / security / billing | 18 /15 /6 /6 /3 |
| Proposed train / validation | 9 /9 |
| Proposed ordinary test / family challenge | 6 /6 |
| Proposed policy shift / retention | 12 /6 |
| Assistant-supervised targets across all records | 5,634 |
The class distribution is intentionally reported rather than called balanced. Billing has only three examples from one family. The QA rule requires the planned classes and limits extreme dominance, but passing that simple bound does not establish representative class prevalence or useful per-class confidence intervals.
The split counts also reveal dependence. Twelve policy-shift records describe only two families under two policies and three surfaces. Treating them as twelve independent real-world incidents would exaggerate the evidence. The synthetic family challenge is a reserved part of this authored universe, not proof of broad operational robustness.
Separate offline containers regenerated the same candidate manifest. The capstone function took approximately 4.11 seconds in one recorded run; this includes generation, reconstruction checks, sealing and the small mask comparison, while excluding container startup and imports. Timing can vary even when bytes agree. A separate audit reconstructed the 48 token sequences and all 5,634 assistant targets without calling the supervision compiler.
Quality is a vector of obligations
Our audit checks schema and identity, label completeness, active-policy consistency, citation, abstention, source eligibility, provenance, coverage, distribution, stable ordering, exact duplicates, near candidates, grouped separation, exposure, deterministic representation, supervision and dataset digests. These checks answer different questions.
For a set of gates $G_1,\ldots,G_k$, admission requires their conjunction:
The figure shows the gates conjoining, the seal binding every artifact to the same audit, and the card regenerated from that audit. Notice that a counterfeit PASS or a reused card fails against the trusted prior seal rather than against its own claims — and that sealing still authorises no freeze.
The negative controls exercise that rule. Revoking the source produces rights-eligibility failures. Recording a proposed test example as a retrieval source fails protected-access QA although the records remain syntactically valid. A supplied quality report saying PASS is insufficient: the seal writer recomputes the audit and compares the result. Altered labels, masks, splits or card bytes fail verification against the trusted prior seal.
The 67 targeted adversarial probes cover these boundaries alongside the broader regression tests. Their passing result establishes the tested software properties. It does not establish legal compliance, population quality, perfect semantic deduplication or independence from the authors’ shared assumptions.
A dataset card explains the artifact to someone else
The motivation for systematic documentation is developed in Datasheets for Datasets. Our card records purpose, origin, generation, permitted uses, rights limitations, personal-data assessment, annotation origin, composition, coverage, splits, policies, duplication method, QA, biases, limitations, reproduction, maintainer and change history.
It explicitly says that there are no actual customer records and no human-adjudicated gold labels. It explains the narrow English templates, the policy pairs and the absence of statistical adequacy. It identifies internal candidate inspection as the current use and excludes production training/evaluation and redistribution.
A card is documentation, not independent validation. Its most useful properties are specificity and consistency. A generic paragraph saying “use responsibly” cannot tell a reviewer which labels changed. Our validator regenerates the expected card from the candidate and QA, then checks its revision and dependency digests. Reusing a well-written card from a different dataset fails that binding.
Seal bytes without pretending to approve them
The candidate directory contains records, annotations, representations, supervision, split proposals, sources, contracts, policies, QA and card, plus a manifest. The manifest binds each file digest and the canonical content, annotation, supervision and split identities. Existing candidate destinations are refused. Verification compares against a trusted previously captured manifest, not a replacement supplied beside edited files.
Hashing is valuable precisely within that limit. Someone able to replace both a file and its purported trusted seal can recompute hashes. Authentication of a producer, reviewer or release authority is a separate service. Our local artifacts make consistency auditable; they do not manufacture authenticated approval.
The original release gate reported READY_TO_FREEZE_PENDING_AUTHOR_APPROVAL. This was a false readiness result: training lacked three classes, identifiers leaked categories, and candidate-supplied approvals were accepted. A bare digest set could even obtain an educational frozen state. Reproducibility did not justify either transition. The exact original artifact is preserved; an external digest-bound disposition now restricts it to PILOT_MECHANISM_ONLY. It is prohibited as a training or protected evaluation benchmark, including when loaded directly through a release API.
The canonical production contract still lacks an approved frozen dataset, fixed acceptance targets, sample adequacy and operational budgets. Its benchmark state remains draft. Those are real prerequisites, not fields to populate with invented measurements. Future changes need a new dataset revision and explicit parent lineage; changing held-out labels in place would invalidate the comparison’s identity.
Reproduce, then decide what the evidence permits
Run the offline CPU capstone to a new destination:
docker compose -f docker-compose.book.yml run --rm book-tests python examples/part_iii/study_demo.py --output /results/data-candidate-NEW.json
Its sibling candidate directory contains the generated artifacts. A separate container can regenerate the candidate and compare manifest identity, then run the dataset verifier against its own report. There are no weights or private source files to recover from Git.
An additional integer-parity task exercises generic provenance, raw record grouping and supervision without inventing incident labels. It also exposes an architectural limit: the current TaskContract schema requires policy-oriented fields even for a policy-free task. Its contract/release step is explicitly blocked, rather than being padded with a fictional routing policy and called general. That schema evolution is future work.
The data pipeline now makes its assumptions and failures inspectable. It has produced a sealed synthetic candidate whose labels, masks and split proposal cannot change unnoticed against its prior identity. It has not produced a production-approved benchmark or evidence that fine-tuning is the right intervention. The next part’s runtime and baseline work must earn those stronger claims under the remaining approvals and resource gates.
Separate technical validation from permission
Revision 0.2 is a new candidate, not a rewrite of the failed pilot. Its pre-generation plan declares 35 mechanism families and 80 records. All five classes have 16 records: four training, two validation, two ordinary test, two composition challenge, four policy-shift and two future retention probes per class. There are ten training families and five in each other role. These are design counts, not evidence of statistical adequacy.
Offline generation and a separate reconstruction counted 9,416 assistant-supervised targets. The earlier 5,634 total remains a property of the pilot alone. The reconstruction reads sealed rows and independently builds token positions, labels and class/role counts without calling the rendering or supervision compiler; it still shares the byte encoder and canonical hashing conventions. Determinism verifies an implementation boundary, not rights or semantic truth.
QA version study-qa/2.0 validates the plan, class coverage, exact permitted input fields, opaque identifier construction, policy pairs, fact-signature candidates, annotation provenance and mask consistency. The card is regenerated from that audit and binds its digest. Verification recomputes current QA against trusted repository resources, so a self-consistent replacement card and counterfeit PASS report cannot promote an artifact. Old QA lacking these gates cannot satisfy the new schema.
The candidate reports VALIDATED_CANDIDATE for technical consistency, RIGHTS_APPROVAL_PENDING for declared uses and NOT_READY_TO_FREEZE overall. Independent semantic review and statistical adequacy remain pending. Its labels are deterministic synthetic references, not adjudicated human gold. Public QA inspection means it is not an untouched final benchmark.
A freeze request needs an independently persisted, authenticated approval bound to the exact manifest, revision, task contract, policies, QA digest, scope, approver, decision and timestamp. The verifier is configured outside the candidate. Bare digest sets, replayed or stale bindings, candidate-generated authority and unverified records are rejected; test-only evidence cannot approve a real release. No genuine approval backend is configured, so the production path remains pending. No freeze was performed.
An integrated negative control combined plausible class counts, valid schemas and recomputed internal hashes with a label-bearing identifier, self-issued rights approval, misleading role allocation and a readiness claim. The technical audit and release verifier refused the respective boundary violations. This is the useful lesson of the pilot: measure what each gate actually establishes before treating a green result as permission to proceed.
Further Reading
- Timnit Gebru et al., “Datasheets for Datasets” (2018). Read the source. Supplies the documentation headings the chapter’s card must satisfy, with matching revision and digests; also recommended in Chapter 13 for the rights ledger.
- Margaret Mitchell et al., “Model Cards for Model Reporting” (2019). Read the source. Shows how to disclose intended uses and benchmarked behaviour across groups, complementing dataset documentation on the model side of the release gate.
- Yizhong Wang et al., “Self-Instruct” (2022). Read the source. Illustrates synthetic generation with filtering, motivating the separation of generator output from independent QA evidence; also recommended in Chapter 15 for format authority.