← Fine Tuning From First Principles

Write the Task Contract Before Training

Define success as a versioned executable contract, separate structural validity from policy correctness, and preserve evaluation rules before training.

“Make the model better at routing incidents” sounds like a requirement until two engineers try to implement it. One counts valid JSON. Another counts correct categories. A third considers a response successful if it sends the incident somewhere plausible. After training, each can report an improvement while disagreeing about whether the system should be used.

The problem is not initially an optimiser problem. The team has not defined the event it wants to improve. Does an abstention count as a safe outcome or an unresolved incident? Does a correct category compensate for an obsolete route? Which policy revision determines the expected action? Without answers, a measurement can change meaning while retaining the same name.

Chapter 02, When Fine-Tuning Beats Prompting, RAG, Tools or Distillation, established that different causes justify different investigations. This chapter defines the object those investigations must share: a versioned task contract. It specifies permitted inputs, the meaning of outputs, the evaluation obligations and the conditions under which evidence could support acceptance. A baseline, an adapter and a release review should consume this same definition rather than quietly inventing their own versions of success.

We will implement a small contract loader and validator in FineTuneLab. It will validate fictional records, distinguish structural validity from policy consistency, reject changes against a trusted frozen reference and keep the intervention planner cautious when evaluation is incomplete. It will not train a model, generate the final dataset or declare the benchmark frozen.

Separate the objective from the thing we can observe

A business objective might be to reduce the time required to get an incident to a responsible team. That depends on intake, classification, policy, queue management and human response. A model that predicts a route influences that objective but does not control all of it. We should therefore distinguish the desired business outcome from the behaviour assigned to this component.

For the running example, the observable behaviour is a structured response to a fictional incident under an identified routing policy. The response contains a category, a severity, a route, an abstention flag and a policy citation. The contract must say what those fields mean, which values are permitted and how insufficient information is handled. Merely listing five field names would establish a wire format, not a task.

Ground truth is another separate object. A schema can establish that severity is one of the permitted strings. It cannot establish that a particular report deserves that severity. That needs a rubric and a trustworthy annotation process, with explicit treatment of ambiguous and contradictory cases. The label definition must precede the model comparison; otherwise the preferred answer can drift towards whichever response the model happened to produce.

Metrics summarise comparisons between observations and reference labels. Acceptance thresholds state which measured values would be sufficient. Operational constraints state which otherwise accurate systems are admissible. Training-data permissions determine whether a proposed learning process may use the inputs at all. Release and rollback rules govern what happens after an evaluation, rather than turning a good score into automatic deployment.

These distinctions prevent a dangerous substitution: satisfying a machine-readable shape is not satisfying a customer, and satisfying an evaluation threshold is not authorisation to release software. The contract makes these boundaries explicit so that later code can enforce the parts it actually understands.

Define the interface before the training representation

The initial input describes a fictional incident through incident_id, text, policy_version, information_status and reported_impact. An asset_id is optional. Unexpected fields are rejected. The identifier locates a record; it is not a clue from which the system may infer the expected label. The text is task data, even if it contains something that looks like an instruction to change the policy.

For these teaching fixtures, information_status is trusted intake metadata with values sufficient, insufficient or contradictory. The validator does not infer that status from prose. A real intake process must justify its availability and accuracy; if it cannot, the input contract must change in a new task version. Similarly, reported impact describes supplied context, not a guarantee that the model has correctly interpreted the incident.

The output vocabulary is deliberately narrow:

Field Initial vocabulary or meaning
category access, service, security, billing, unknown
severity low, medium, high, unknown
route A team in the active policy or human_review
abstain A JSON boolean indicating a refusal to make the automated classification/routing decision
policy_citation Exact policy identity and revision, such as triage-policy:p1

The categories separate credentials/permissions, service availability, suspected compromise and billing. unknown represents an unresolved classification, not a fifth ordinary business department. Severity describes impact and urgency under a rubric: ongoing outage or credible compromise is high; multiple-user degradation is medium; a single-user non-security issue is low. Insufficient or contradictory information requires unknown labels and human review. Overlaps, such as an access problem with evidence of compromise, need adjudication under that rubric rather than whichever keyword occurs first.

This vocabulary keeps the example inspectable and exposes important failure boundaries. It is not a validated taxonomy of real support operations. Another organisation may need different categories and severity definitions; that is a different task definition, not a reason to reuse the same metric name with changed labels.

Our abstention rule requires abstain=true, category="unknown", severity="unknown" and route="human_review" together. The citation still identifies the policy containing the escalation rule. A sufficient intake may also be voluntarily escalated, but coverage and the treatment of high-severity misses must make that choice visible. Returning human_review with abstain=false, or claiming to abstain while supplying a confident category, is inconsistent under this contract.

Malformed inputs produce structured rejection diagnostics. They are not silently coerced into valid incidents. If an application later wants a user-facing repair or escalation flow, that wrapper needs its own defined behaviour; the validator does not manufacture an apparently valid model output from an invalid request.

Keep stable labels separate from current policy

Imagine an access incident involving a fictional user’s credentials. Under policy p1, an access category maps to access_team. Under policy p2, the same category maps to service_ops. The report’s category and severity need not change. Its correct route and citation do.

The demonstration preserves the stable output labels and changes only the active policy, the route and the citation. Under the revised contract, the old output fails policy consistency while the updated output passes it. This is a controlled illustration of a policy-dependent decision, not a model learning experiment.

The contract records policy identity, version, source and a hash of the parsed policy content. The task-specific semantic validator checks the supplied policy against that identity before using its table. A file called “current-policy.json” is not sufficient provenance: its contents can change while the name stays the same. Conversely, a plausible citation string is not enough if the loaded table does not match the expected hash.

There is an important limit to this check. A route consistent with a predicted category can still be wrong because the category is wrong. Looking up the billing route for a misclassified access incident does not fix classification. Our validator can check the declared interface and the policy relation; it cannot establish that the free-text incident was understood correctly. That requires independently labelled evaluation examples and an appropriate scorer.

Three kinds of validation

FineTuneLab separates structure, task semantics and readiness. Structural validation checks types, required fields, enumerated values and unexpected fields. We use the JSON Schema 2020-12 validation vocabulary through a specifically selected validator. The project supplies a deliberately small scalar-record schema surface; it does not promise support for arbitrary multimodal or nested task records at this stage.

Cross-field contract checks go further than JSON shape. They reject duplicate metric identities, contradictory directions, undefined target references, invalid split use and incompatible operational requirements. “Operate offline” and “require the live network policy” cannot both be satisfied. Claiming a benchmark freeze while dataset identities and adequacy requirements are missing is also inconsistent.

Task semantics live behind a separate SemanticRules interface. The incident implementation checks policy identity, route/category consistency and the abstention relationships. Another task can supply a different semantic implementation without teaching the generic contract loader about incident categories. Structural success is checked before task rules are called, so a missing field does not become an incidental dictionary exception deep inside policy code.

Readiness asks whether the specified prerequisites for a benchmark are present. A valid draft may intentionally contain null dataset hashes, provisional acceptance targets and unresolved sample adequacy. It should load so that people and software can inspect those gaps. It must not become ready merely because its JSON is well formed. The reference contract reports its gaps and keeps verified acceptance at NOT_EVALUATED.

This separation is useful during development: an incomplete but honestly represented plan differs from an internally contradictory contract, and both differ from a model that failed its evaluation. Collapsing them into a single “valid” flag would conceal the reason training should wait.

Three validation layers, one contract

The figure stacks the three gates in the order they run and points each executed validator outcome at the layer that produces it. Notice that passing an earlier layer never settles a later one: a well-formed record can still violate task semantics, and a consistent record can still leave the benchmark unready.

Choose a primary metric and keep the other obligations visible

For this reference task, the provisional primary metric is routing accuracy under the active policy. The component exists to choose the action that reaches the responsible team. Category macro-F1 and severity accuracy diagnose the stable classification contribution. Schema validity is a gate; a route inside an unusable record does not meet the interface. High-severity miss rate is safety-critical, and retention performance checks behaviour outside the adapted task.

Write the routing metric as

\[ \operatorname{RouteAccuracy}=\frac{1}{n}\sum_{i=1}^{n} \mathbf{1}\{\hat r_i=r_i^*\}. \]
The denominator includes all evaluated incidents; an invalid output is a failure. Expected human-review actions can be correct routes, so routing accuracy alone does not show how much work the system automated. That is why abstention needs its own view.

For a known category \(k\), define

\[ F_{1,k}=\frac{2TP_k}{2TP_k+FP_k+FN_k},\qquad F_{1,\mathrm{macro}}=\frac{1}{4}\sum_{k\in K}F_{1,k}, \]
where \(K\) contains the four non-unknown categories. Every category receives equal weight, so a frequent category cannot dominate the average merely through its count. Invalid or abstained predictions contribute a false negative for the known reference category. The contract makes the otherwise easy-to-miss zero-denominator convention explicit: if a class term is undefined, report the macro score undefined rather than silently removing that class. The [scikit-learn F1 reference](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html) illustrates why averaging and undefined-value policies must be specified; this chapter does not call its scorer or report a measured F1.

Coverage is the fraction of incidents receiving a valid non-abstaining response. Selective risk is routing error among those responses:

\[ \operatorname{Coverage}=\frac{|A|}{n},\qquad \operatorname{SelectiveRisk}=\frac{1}{|A|} \sum_{i\in A}\mathbf{1}\{\hat r_i\ne r_i^*\}, \]
where \(A\) is the set of valid, non-abstaining outputs. Selective risk is undefined if \(A\) is empty. A system that abstains on every case does not obtain zero selective risk by default. A risk reduction accompanied by sharply reduced coverage describes a trade-off, not a free improvement.

The initial high-severity miss definition treats a below-high prediction, invalid output or abstention on a gold high-severity incident as a miss, with abstentions also reported separately. This conservative definition describes what the automated component failed to resolve; it does not claim that human escalation is itself harmful. An end-to-end measure incorporating the outcome of human review would be another metric and would need new evidence.

Policy-citation validity, severity accuracy and retention agreement have their own specified denominators. Latency is an operational metric whose workload and aggregation protocol still need to be fixed. These measures are not equally important and need not all increase together. A narrower model may improve routing but damage retention. An aggressive abstention rule may lower selective risk while leaving the business objective unmet. An output constraint may increase valid JSON without improving category discrimination.

The formulas here specify quantities. We have not computed them on model predictions, implemented the complete scorer or established statistical reliability from the few teaching cases. The contract stores the definitions so that those later implementations can be checked against an agreed target.

A target is not yet a frozen threshold

Keep five states separate. A metric specification defines the quantity. A provisional acceptance target proposes an adequate value or effect. A frozen threshold commits an experiment to that rule. An observed measurement is produced by an executed evaluation. A verified acceptance result checks the measurements and their evidence against the frozen rules.

The architecture’s proposed targets remain provisional in the reference contract: schema validity at least 0.99; a routing-accuracy improvement of at least 0.03 over an admissible control; retention degradation no worse than 0.02; and no increase in high-severity miss rate. The primary effect also has a planned uncertainty requirement. These numbers are design proposals inherited from planning, not empirical findings or newly approved release criteria.

For an effect \(\Delta=\operatorname{RouteAccuracy}_{\mathrm{candidate}}- \operatorname{RouteAccuracy}_{\mathrm{control}}\), a future acceptance rule may require both the minimum effect and an uncertainty bound above zero. A point estimate alone does not establish that requirement. The contract names a paired family-bootstrap approach as planned work, with its protocol and implementation still outstanding. We do not invent confidence intervals or sample counts to fill the gap. Chapter 38 will develop the statistical treatment, while Chapter 18 must establish the adequacy and evaluation protocol needed for the actual benchmark freeze.

Resource constraints have the same discipline. The 12,288 MiB GPU target describes a hardware aspiration, not measured training fit. Latency and inference-cost limits remain null until appropriate budgets are established. Model and tokeniser revisions, template behaviour and target masking must be verified before a real run. A positive budget cannot prove compatibility, and a contract that names these obligations has not performed them.

Dataset identity is part of what a result means

The contract names a planned dataset and six intended boundaries: training, validation, ordinary test, family challenge, policy shift and retention. Their uses are explicit. Training examples supply updates; validation supports model and prompt selection; final evaluation sets are not sources of training examples or iterative tuning feedback. Retention probes establish a separately identified capability requirement.

Each future split needs an immutable manifest reference and content hash, along with label provenance, policy compatibility and an access record. The reference contract uses null manifest identities because no complete audited dataset has been generated. A fabricated hash would make the file look complete while supplying no evidence. Chapters 13–18 must implement these requirements and freeze the actual identities.

Group separation applies to scenario families, templates and entity groups, not merely unequal record IDs. A new identifier attached to a paraphrase of a training template does not establish a meaningful generalisation challenge. The generator and audit must make these groups explicit and test their separation. Policy-shift cases also need a known policy revision so that a changed expected route is not mistaken for an inconsistent label.

Test access must be recorded because observing held-out errors can influence later decisions. If those observations become tuning guidance, the original claim of an untouched final evaluation no longer holds for the subsequent experiment. Version the new study instead of rewriting the previous result’s history. The same boundary applies to examples chosen for prompts, teacher-derived training records and preference pairs.

Version identity, compatibility and a trusted freeze

The schema version identifies the contract format a reader understands. The task version identifies a particular task definition. Policy and dataset versions identify other moving parts. These are different identities: a policy revision may preserve the record shape while changing the correct action. Wire compatibility does not make two evaluations interchangeable.

We use a three-component task version and require changed content to receive a greater version plus a reference to the previous contract hash when validated as a revision. The Semantic Versioning specification is useful vocabulary for interface changes, but our evaluation rule is stricter than an ordinary API-compatibility claim: even a structurally compatible scoring change defines a different comparison. Do not pool results across contract hashes without an explicit cross-version study.

The content hash covers a project-defined deterministic JSON encoding, excluding the hash field itself. This convention sorts object keys and rejects non-finite numbers; it is not presented as a general standards-compliant JSON canonicalisation scheme. The loader also rejects duplicate JSON keys, preventing a critical field from being silently replaced during parsing.

A hash is an identity check, not approval. Someone who edits a file can recompute its hash. To detect a silent change to a frozen version, a caller must retain a trusted prior contract and validate the candidate against it. Our test deliberately changes an acceptance target and recomputes the hash; the same-version change is still rejected as FROZEN_MUTATION. A legitimate revision needs a new task version and explicit parent identity. The library does not authenticate the person approving that revision or implement an external trusted registry.

The demonstration creates an in-memory ILLUSTRATIVE frozen copy solely to test that guard. The repository’s reference contract remains DRAFT, with provisional targets and an unfrozen dataset. Declaring a BENCHMARK freeze while required manifests, adequacy or resources are incomplete is rejected. Even complete-looking metadata would still need an actual audit of the referenced evidence; the library neither downloads datasets nor certifies a release.

Run the contract through its failure paths

The authoritative format is packaged at examples/ftlab/src/ftlab/schemas/task-contract-1.0.json; the initial contract and policy are in examples/ftlab/contracts/. The reusable loader is ftlab.contracts, while incident-specific semantics are in ftlab.triage. The structural implementation uses the optional jsonschema dependency, verified locally at version 4.26.0. The original decision-only component remains standard-library compatible.

From the repository root, run:

python -m examples.ch03.contract_demo --output experiments/FT-03/my-run.json
python scripts/run_tests.py --output .local/ch03/my-test-timings.json

Choose new report paths. In a separate environment that needs the contract dependency, install the project’s optional contract extra; no language-model packages or weights are required for this example. The jsonschema validator documentation distinguishes validating a schema itself from validating records against it. FineTuneLab selects the 2020-12 validator explicitly and adds its own cross-field checks.

The complete executable demonstration loads the contract, validates the fictional input and output, and produces specific failures:

Demonstration Observed validation outcome
Correctly shaped access case under policy p1 No structural or supplied policy-rule diagnostics
Invented category STRUCTURE_ENUM
Incident policy revision differs from the contract POLICY_VERSION
Insufficient information without abstention ABSTENTION_REQUIRED
Consistent human-review abstention No diagnostics
Same stable labels, obsolete route/citation under p2 POLICY_ROUTE and POLICY_CITATION
Same stable labels with the revised route/citation No diagnostics
Rehashed target change under a trusted frozen version FROZEN_MUTATION

These are executed validator outputs on constructed records. They do not show how often a model would satisfy the contract. In particular, the policy validator checks consistency with the predicted category, not the truth of that category. The output remains NOT_EVALUATED for verified acceptance and NOT_RUN for model execution.

The tests also cover unknown fields and schema versions, invalid severity, contradictory constraints, metric identity/direction, split usage, hash tampering, revision lineage and an incomplete benchmark freeze. Every test is timed. Known tests strictly over 60 seconds are omitted from routine chapter checks and reported as NOT_RUN_LONG; threshold and selection tests use controlled clocks rather than slow workloads. Discovery/import time is reported separately from suite execution.

The bridge to Chapter 02 takes the task identity, output obligations and constraints from the contract rather than accepting duplicate caller overrides. Observations such as failure cases and declared capability remain separate. The draft contract’s missing evaluation prerequisites cause ABSTAIN; its mere existence does not make a method successful. Future baseline, training and release code should retain the contract hash with its evidence and consume this same authority.

The observation adapter has an explicit v1 mapping boundary. Legacy observations remain valid, while an optional schema-version tag and budget-proposal identifier can be recorded without deriving the envelope from every field of the planner’s current dataclass. Tests cover a future optional defaulted planner field. Sealing a contract also reports non-finite values through the same structured ContractError family as loading them; ordinary canonical hashes are unchanged.

What this contract can establish

We can now detect malformed records, inconsistent policy-dependent fields, incomplete prerequisites and silent changes against a trusted prior version. We can identify the task definition that a future result claims to measure. We have not established the quality of a model, the correctness of every label, the suitability of a sample size or the reliability of human escalation. Those require evidence that no schema can supply by itself.

This is also why release criteria and rollback criteria appear in the contract without a deployment implementation. A later release must bind the accepted model or adapter, policy, scorer, dataset and contract identities, and preserve an explicit prior accepted target. A changed policy or scorer can invalidate an earlier acceptance claim even if the adapter file has not changed. The current library records these obligations; it does not perform a release or rollback.

Before editing the example, choose a requirement that would change its meaning: allow a new category, alter the high-severity miss convention or change an acceptance target. Identify the affected labels, metric definition, policy or evidence. Then attempt the same-version edit against an illustrative frozen reference and inspect the rejection. Finally create an explicitly versioned revision. The exercise is about preserving what a result means, not finding a way around the guard.

Status: TECHNICALLY_CHECKED_WITH_LIMITATIONS; author accepted with limitations; unpublished draft. The loader, schema checks, incident semantics, revision guard and planner bridge have executed on fictional examples. Targeted review fixes have separate evidence under experiments/FT-03-R1/; original records are preserved. The benchmark, dataset, adequacy protocol and acceptance thresholds remain unfrozen; no training or model evaluation has run. Chapter 04, Budget the Adaptation, budgets a defined intervention against an explicit task rather than an undefined aspiration to make a model better.

Further Reading

  1. A. Wright, H. Andrews and B. Hutton (eds.), “JSON Schema Validation: A Vocabulary for Structural Validation of JSON” (2022). Read the source. The normative specification behind the contract’s structural layer, fixing what type, enum and required keywords actually assert.
  2. Scikit-learn developers, “sklearn.metrics.f1_score” (living documentation). Read the source. Specifies the averaging and undefined-value conventions the contract must pin down, clarifying the book’s stricter macro-undefined rule.
  3. Tom Preston-Werner, “Semantic Versioning 2.0.0” (specification). Read the source. Supplies the versioning vocabulary the contract borrows, while the chapter explains why evaluation identity must be stricter than API compatibility.