← Fine Tuning From First Principles

FineTuneLab: Release, Runbook and Decision Handbook

Follow FineTuneLab from installation to a rehearsed rollback with commands that were actually run, then collect the operating runbook, the limits of the release rehearsal and a decision handbook that separates what the code decides from what a person must judge. Fine-tuning is an engineering decision supported by evidence.

The previous chapter, “FineTuneLab: Run and Compare the Training Pipeline”, ran every branch of the pipeline and reached a decision on each. None of the decisions was to release a trained model. That is not a failure of the pipeline. It is what the evidence supported.

This final chapter turns the pipeline into something a reader can operate. It follows the journey a user takes, from installing the package to rehearsing a rollback, with commands that were run and outputs that were observed. It then gathers the operating knowledge into a runbook, states exactly what the release rehearsal does and does not establish, and closes with a decision handbook for the next dataset. The question for the chapter is practical: how does a reader use FineTuneLab, and how do they decide what to do with what it tells them?

FineTuneLab is an educational, offline, CPU-only tool. It is not a production platform, and nothing in this chapter should be read as instructions for deploying a model.

The reader’s journey

The journey has ten stages: install, configure, audit, plan, run, evaluate, compare, decide, rehearse a release and verify recovery. Each stage below shows the command and what it produced. Output lines are verbatim apart from directory prefixes.

    flowchart LR
    I[install wheel] --> C[configure: 6 fields]
    C --> A[audit: NOT_RUN_AUTH for incident]
    A --> P[plan: freeze]
    P --> R[run: check currency]
    R --> E[evaluate: report]
    E --> D[decide: DO_NOT_TRAIN etc]
    D --> H[rehearse release: docket demo]
    H --> V[verify recovery]
  

1. Install

FineTuneLab is distributed as a wheel, ftlab-0.12.0-py3-none-any.whl. The integrated wheel declares jsonschema==4.26.0 as a mandatory runtime requirement. It does not bundle the numerical environment, so compatible Python, Torch, NumPy and safetensors must already be provisioned. The verified installation creates a virtual environment, installs the wheel without consulting any index, removes the source checkout from the import path and runs from a neutral directory:

python -m venv --system-site-packages /results/reader-env
/results/reader-env/bin/python -I -m pip install --no-index --no-deps --no-compile ftlab-0.12.0-py3-none-any.whl
unset PYTHONPATH PYTHONHOME
cd /tmp
/results/reader-env/bin/ftlab --help

The original authoring environment first lacked jsonschema. Integration corrected the wheel metadata and verified offline dependency resolution in a separate virtual environment without system packages. The reader commands below use the pre-provisioned pinned runtime; --no-deps is appropriate only after those requirements have been installed.

2. Configure

A configuration is six fields. For the incident profile:

{
  "schema_version": "1.0",
  "profile": "incident",
  "seed": 37,
  "steps": 8,
  "output_id": "reader-incident-run",
  "backend": "rule"
}

“FineTuneLab: Design the Usable Solution” explained every field and every refusal. The configuration cannot grant approval, name a dataset or select a model.

3 and 4. Audit and plan

$ ftlab audit --config incident.json --output incident-audit
{"command": "audit", "output": "incident-audit", "status": "COMPLETED_EDUCATIONAL_ONLY"}
$ ftlab plan --config incident.json --output incident-plan
{"command": "plan", "output": "incident-plan", "status": "COMPLETED_EDUCATIONAL_ONLY"}

Neither command trains, loads a model or reads data beyond the configuration and the installed contract. The audit records that production training is NOT_RUN_AUTHORIZATION because the incident contract forbids it. The plan freezes the configuration, source hashes, environment, task identity, budget and the three-node graph.

5. Run

$ ftlab run --config incident.json --plan incident-plan/plan.json --output incident-run
{"command": "run", "output": "incident-run", "status": "COMPLETED_EDUCATIONAL_ONLY"}

The run refuses to start unless the plan is current for this configuration, source and environment.

6. Evaluate

$ ftlab report --run incident-run --output incident-report
{"command": "report", "output": "incident-report", "status": "COMPLETED_EDUCATIONAL_ONLY"}

Evaluation here means verification and recomputation. The report checks every seal and receipt, recomputes the decision from the raw observations and confirms that the plan is still current. It does not judge whether the decision is wise.

The same sequence for the tiny-copy profile, with seed 37 and 8 steps, trained the tiny model for 24 updates in 5.1 seconds of wall-clock time for the whole command, including Python and Torch start-up, and reported KEEP_BASELINE: 0/16 against 0/16 on the held-out rotation task, and export UNSUPPORTED_BACKEND. The baseline’s mean NLL was 4.563783884048462. The adapted model’s was 4.559067726135254, equal to the integrated record in this pinned CPU environment. A different Torch build may change floating-point values, and PyTorch’s reproducibility notes warn that results are not guaranteed to be identical across platforms. The decision, the counts and the outputs were identical. A reader who reproduces the run should expect the same.

7. Compare

compare takes two token-evaluation envelopes. No ftlab command writes them: the caller assembles each one as a JSON object with the fields task, data_digest, evaluator, split and predictions, where each prediction has id, expected_ids, generated_ids and exact. For this chapter the two envelopes were assembled by hand from the baseline and adapted heldout predictions of a 2-step tiny-copy demo, with a descriptive label in place of a data hash in data_digest. Provenance of an envelope is the caller’s responsibility; the command checks only that the two envelopes are comparable and internally consistent.

$ ftlab compare --left left.json --right right.json --output cmp-ok
{"command": "compare", "output": "cmp-ok", "status": "COMPLETED_EDUCATIONAL_ONLY"}

The comparison was {"left": 0, "right": 0, "denominator": 16, "effect": 0.0}. A train-split envelope or a different evaluator is refused with INCOMPARABLE_TASK; an exact-match flag that disagrees with the generated tokens is refused with TEST_ACCESS. The command compares token predictions on one task. It does not rank workflow results or combine incident, preference and recovery metrics.

8. Decide

The report directory holds decision.json and report.txt. For the incident run, report.txt contains the decision:

FineTuneLab educational report
{
  "recommendation": "DO_NOT_TRAIN",
  "reason": "Training forbidden by canonical draft contract; useful rule-control report remains available",
  "training_success_claim": false
}
Authenticated execution: NOT_ESTABLISHED
See result.json for raw observations and blocked branches.

The decision is the tool’s recorded output for that profile, not an acceptance. Deciding what to do with it is the reader’s step, and the handbook at the end of this chapter separates what the code decides from what a person must judge.

9. Rehearse a release

The docket profile is the only profile with a release path, because it is the only one with a supported export. One command runs it:

$ ftlab demo --profile docket --steps 2 --output demo-docket
{"command": "demo", "output": "demo-docket", "status": "COMPLETED_EDUCATIONAL_ONLY"}
$ ftlab report --run demo-docket/execution --output report-docket
{"command": "report", "output": "report-docket", "status": "COMPLETED_EDUCATIONAL_ONLY"}

The demo writes its configuration and plan first, then runs. Its decision is KEEP_RULE_BASELINE, with task scores 0.333 for the base system, 1.0 for the constructed system and 1.0 for the rule. The rehearsal inside it is the subject of the next section.

10. Verify recovery

The recovery profile interrupts and resumes training on a small classifier:

$ ftlab demo --profile recovery --steps 2 --output demo-recovery
{"command": "demo", "output": "demo-recovery", "status": "COMPLETED_EDUCATIONAL_ONLY"}

Its decision is COLLECT_MORE_EVIDENCE, because it demonstrates a mechanism on a separate task and ranks no model. The book’s records show the resumed run equal to the uninterrupted run, step by step, at both seeds.

The release rehearsal

Record FT-P8-INTEGRATED-02-P8-48A-s37 holds the rehearsal, and the docket demo above reproduced every count below. It reuses the Part VII sandbox described in “Operating, Auditing and Rolling Back a Fine-Tune”.

The constructed docket system is exported, reloaded against its manifest and served on the loopback interface. Twenty-four HTTP requests all return status 200, and every response’s output and probability vector equals the library’s output for the same case. Two releases, v1 and v2, are registered in an in-memory sandbox and activated in turn. A fault then substitutes a different model under the v2 label, bypassing the controller. Health becomes false: the runtime model’s stored identity no longer equals the expected identity. The substituted model’s outputs differ from the baseline on 12 of the 24 cases.

Rollback is attempted and interrupted before commit; nothing is committed and v2 stays active. The rollback is retried and commits, activating v1, and is recorded as not idempotent, because it changed state. It is repeated, and the repeat is recorded as idempotent. After rollback, all 24 outputs equal the baseline outputs and the model identity matches v1’s. The audit log has 14 events, each linked to the digest of the one before.

The authoring cross-check recomputed these facts from the raw record: the 24 response bodies against the baseline outputs, the 12 differing regressed outputs, the 24 restored outputs, the hash links of all 14 audit events and the rollback and health flags.

What the rehearsal does not establish

The rehearsal establishes behavioural restoration on 24 declared cases. Five limitations, recorded in the evidence itself, bound everything else.

Stored identity is not live state. The health check compares the runtime object’s stored identity with the expected identity. It does not rehash live tensors, so an in-place change to a tensor that left the stored identity unchanged would not be detected. The export path is different: the export manifest records a weight identity computed from the exported weights, so a mismatch between exported and declared weights is caught at load time. A health check that recomputes the weight identity from live memory would be stronger, and it is not implemented.

One approval authority. The sandbox combines the review and authorisation roles in one simulated capability. Its audit shows REVIEW_APPROVED and RELEASE_AUTHORISED with scope SIMULATED_NOT_PRODUCTION, while the release manifests themselves still record review as PENDING and release_authorised as false. There is no independent human approval anywhere.

The fault is not in the audit. The model substitution bypassed the controller, so the audit trail contains no event for it. An audit log records what passes through the controller, not everything that happens.

No durable recovery. Rollback state lives in memory. A process crash, a power loss or multiple workers are outside the implementation, and there is no journal to replay.

Shared oracle. The constructed system and the rule share the policy that defines gold, so their 24/24 is agreement by construction.

Google’s chapter on release engineering describes the practices a real release needs, including hermetic builds whose output does not depend on the build machine and packages labelled with build identifiers so that a specific release can be reconstructed. FineTuneLab’s plans and receipts borrow the identification discipline. They do not provide the rest, and the real deployment proposal, P8-48B, is NOT_RUN_AUTHORIZATION.

Runbook

Install. Use a Linux environment with Python 3.13 and the dependency versions the plan records; the book’s runs used Torch 2.6.0 built for CPU and NumPy 2.5.5. Install the wheel with --no-index --no-deps, unset PYTHONPATH, and run from a directory outside any source checkout. Native Windows is not a verified runtime.

Environment. A plan binds the Python version, the Torch build, the platform, the device and the thread count. A different build produces a different plan. Do not try to run another environment’s plan; make a new one.

Backends. incident/rule, docket/constructed-linear, tiny-copy/tiny-transformer, preferences/objective-arithmetic and recovery/resume-net. No other pairing exists. No pretrained model, user dataset or GPU can be selected.

Permissions. Training is allowed only for tiny-copy and recovery, on the book’s fixtures, under the recorded author directive with no production authority. The incident contract forbids training, and no configuration can change that.

Configuration. Six exact fields; seeds 37 or 38; steps 1 to 16, and at least 2 for recovery; a simple output identifier; at most 32,768 bytes; no duplicate keys or non-finite numbers.

Demo. ftlab demo --profile PROFILE --output NEW_DIRECTORY writes a configuration and plan, runs, and places the result under execution/. The default profile is docket, the default seed 37 and the default steps 8.

Evidence. Each run directory holds plan.json and result.json. Keep them together and unedited. A seal detects change; it does not prove who ran the command, and authenticated_execution is always NOT_ESTABLISHED.

Results. report writes report.txt and decision.json. Read result.json for the raw observations and the blocked branches.

Failure codes. The codes a reader is most likely to meet are these. Configuration: CONFIG_FIELDS, CONFIG_JSON, CONFIG_SIZE, CONFIG_PATH, TASK_UNDECLARED, SEED, TRAIN_BUDGET. Authority and support: UNSUPPORTED_BACKEND (exit status 3). Currency: PLAN_CURRENT. Output: OUTPUT_EXISTS. Comparison: INCOMPARABLE_TASK, TEST_ACCESS, COMPARE_CASES. Verification: RESULT_BINDING, DECISION_BINDING, GRAPH_RECEIPTS, GRAPH_TASK, GRAPH_TYPE, GRAPH_EDGES. Budget: STAGE_BUDGET. Record integrity: EVIDENCE_IDENTITY, raised when a sealed record has been edited; plan structure: PLAN_SCHEMA, GRAPH_IDS. Internal checks such as HTTP_EQUIVALENCE and ROLLBACK_STATE guard the docket rehearsal. Every failure other than an authorisation or unsupported-backend refusal exits with status 2. Two kinds of failure do not produce a named code: a mistyped command line gets argparse’s usage message, and a missing file gives the operating system’s message in place of a code, as in the interruption case below. Preserve the failed output and the error line; do not edit records to make a check pass.

Interruption. The command has no resume. A tiny-copy run interrupted with SIGINT after its plan.json appeared left a partial directory containing plan.json, but no result.json. Reporting on it failed with a file-not-found error, and rerunning into it was refused with OUTPUT_EXISTS. Start again in a new directory and keep the partial one as a record. Exact resumption of training state is a library mechanism, demonstrated by the recovery profile, not a feature of run.

Resource limits. The plan declares CPU only, one thread, no network, 180 seconds per stage and at most two seeds, with tiny-copy allowed three optimiser updates per configured step across its three stages (at most 48 at 16 steps; the book’s runs used 24) and recovery twice the steps. These are declared and partly checked. The seed and step bounds are refused at configuration time. The stage time is checked after a profile finishes and per training step, not by an external timer that would stop a runaway stage. The update counts follow from the code rather than being checked against the plan field. There is no memory quota. The docket rehearsal makes loopback HTTP requests, so “no network” means no external network.

Shutdown. Each command runs to completion and exits; nothing stays running. The docket rehearsal starts a loopback HTTP server for its requests and stops it within the run. No public port is bound.

When not to fine-tune. When the contract forbids it, when a non-training baseline matches the best candidate, when the remaining error is not in the model, when the candidate improves loss but not the task, and when the result cannot be exported, served and rolled back. The handbook below makes this precise.

What was verified, and how

The engineering delivery reports its test results in separate categories, and they should not be merged. Forty new tests were added for Part VIII. The final full suite ran 1,455 tests: 1,454 passed and 1 failed. The failure is inherited: it is a Part VI integration test that checks that recorded Part V remediation source files are unchanged, and two such files differ from its expectation. It is reproduced, not caused, by Part VIII and is not modified here. The suite is therefore not green. The installed-wheel gate, which builds, installs and exercises the package outside the checkout, passed and is classified as long because it has taken more than 60 seconds. The engineering delivery also ran 268 independent numerical checks. Separately, the authoring cross-check for these chapters recomputed the central numbers from the raw records with the standard library alone and is reported in the evidence package; it does not count the engineering checks as its own.

Decision handbook

The handbook separates two kinds of guidance. The first is logic the code implements: a decision computed from recorded observations, which a reader can rerun and audit. The second is guidance this book recommends, which no code enforces and which requires a person’s judgement.

Implemented logic

Most of the implemented decisions are fixed per profile. The code does not weigh evidence for them; it returns the recorded outcome for that profile, with a reason that names the evidence the author judged decisive.

Input What the code does
incident profile Always returns DO_NOT_TRAIN, with no training-success claim; the recorded reason is the contract’s training_allowed = false
docket profile Always returns KEEP_RULE_BASELINE, with the base, constructed and rule task scores attached; it does not compare them
tiny-copy profile, paired exact-match effect zero or negative KEEP_BASELINE
tiny-copy profile, paired exact-match effect positive COLLECT_MORE_EVIDENCE (the only comparison the decision function makes)
preferences or recovery profile Always COLLECT_MORE_EVIDENCE
Plan, source, environment or configuration changed Refuse with PLAN_CURRENT
Profile paired with the wrong backend Refuse with UNSUPPORTED_BACKEND
tiny-copy export Recorded as UNSUPPORTED_BACKEND in the result; the run still completes
Every run Lists real-model, GPU and production branches as NOT_RUN_AUTHORIZATION; no configuration can request them

No implemented decision recommends a release. Even a positive effect yields only a request for more evidence.

Runbook decision tree separating code logic from human judgement

The figure puts fixed code outcomes beside the eight human criteria for a justified fine-tune. Any negative answer keeps the rule or baseline with written reopening conditions. It describes the educational release rehearsal (stored identity, in-memory sandbox); real deployment stays not run.

These recommendations come from the book’s argument and experiments. They are not encoded in FineTuneLab.

  1. Write the task contract first. Define the task, the correct output, the permitted data and whether training is allowed, as in “Write the Task Contract Before Training”.
  2. Measure the strongest alternative. Prompting, retrieval, tools, rules and distillation each change a different thing. Measure them on the task before proposing an update to weights.
  3. Find where the error lives. If it is in permissions, tools, post-processing or the evaluation itself, training will not fix it.
  4. Fix the evaluation before training. Split by family, freeze validation and test, declare slices, weights, comparators and a practical threshold.
  5. Decide on task success, not loss. Read the outputs. A constant output and a lower loss can coexist.
  6. Report every slice and every retained skill against named comparators.
  7. Require a release path with export, identity checks, monitoring and a rehearsed rollback before calling a candidate deployable.
  8. Write down the reopening conditions whenever the answer is not to fine-tune.

The ML Test Score of Breck and colleagues offers a complementary rubric for production readiness, and model cards give a format for reporting a model’s performance across groups and its intended contexts. Both assume a model worth releasing. The handbook’s first job is to establish whether there is one.

What remains open

The book’s own acceptance criteria call for a LoRA fine-tune of a pinned open model, executed end to end, and a fixed benchmark that compares prompting, retrieval or tools, and trained methods with raw predictions and uncertainty. Neither was authorised in this part. Both remain NOT_RUN_AUTHORIZATION, and they are blocking limitations of the book’s evidence, not completed work. The tools to run them responsibly exist. The permission, the rights and the compute do not.

Closing: the first-principles questions

The first chapter, “What Fine-Tuning Actually Changes”, ended with five questions to ask of any proposed intervention. Forty-seven chapters later, they can be answered with evidence rather than intuition.

What changes at request time? A prompt, a retrieved passage or a tool result changes the input. Training changes none of those; it changes the function that receives them. The incident rule shows that much of what a team wants from a model can be done at request time.

Which persisted parameters can training modify? Exactly the ones the run declares: all of a tiny model’s parameters in the source and continuation stages, a rank-2 adapter on blocks.0.project in the adaptation stage, or none. The adapter manifest and the recorded identities make that a fact rather than an assumption, and the library’s compatibility check refuses adapter targets that were never verified, although the command-line path does not call it.

Where does the supervision come from? From a contract and its policy, through masks that decide which tokens are targets, and from traces that must be selected before they are trusted. Supervision that encodes failures teaches failures.

What information must remain current? Policies change, as the p1 and p2 revisions showed. Information that changes belongs in a source that can be updated at request time, not in weights that must be retrained.

What evidence would justify carrying the extra trained artefact into operation? Permission, a measured and learnable gap over the strongest alternative, task-level success on a fixed and separated evaluation, no unacceptable regression on any slice, a working release and rollback path, and a cost worth paying. In this book’s experiments, that evidence did not arrive. The tiny model’s loss fell and its answers did not change. The rule matched the adapter. The contract forbade training. Each time, the pipeline said so.

That is the book’s conclusion, and it is not a pessimistic one. Fine-tuning is a powerful tool, and every chapter has shown how to use it carefully. Fine-tuning is an engineering decision supported by evidence, not a default response to every model limitation. A team that can measure the alternatives, locate the error, fix the evaluation, read the outputs and roll back the result will know when to fine-tune. More often than expected, it will also know when not to.

References

Further Reading

  1. Margaret Mitchell et al., “Model Cards for Model Reporting” (2019). Read the source. The reporting format for any model that ever passes the handbook’s release bar. Also recommended in Chapter 18 for the model side of the release gate.
  2. Timnit Gebru et al., “Datasheets for Datasets” (2021 journal version; preprint 2018). Read the source. The dataset-documentation standard marking the boundary of the book’s synthetic-only evidence. Also recommended in Chapters 13 and 18.
  3. PyTorch contributors, “Reproducibility” (living documentation). Read the source. The upstream non-guarantees behind plan currency and the limits of rerun reproduction.