FineTuneLab: Design the Usable Solution
Describe the installed ftlab command as an architecture: a strict configuration, fixed profile and backend pairs, contracts that cannot be overridden, a three-node typed graph with sealed receipts, and separate meanings for planning, authorising, executing, evaluating and accepting. Every command and output shown was run.
Forty-five chapters have built FineTuneLab one module at a time. Each chapter called its own functions from its own experiment driver, and each experiment lived in a namespace of its own. That is a good way to build evidence and a poor way to use a tool. A reader who wants to try the workflow on the book’s fixtures should not need to know which chapter’s driver to import, which seed the experiment used or which receipt binds which artefact.
This chapter describes the interface that closes that gap, the installed ftlab
command, and the design decisions behind it. The central question is
architectural: how does a usable interface stay as strict as the experiments it
wraps? A command-line tool is where convenience usually wins. Defaults appear,
flags multiply, a configuration field is added that turns a check off, and the
evidence discipline of the library leaks away through the front door. The design
here is built to prevent that, and the chapter shows the refusals as well as the
successes. Sculley and colleagues described how machine-learning systems
accumulate hidden technical debt,
maintenance costs that stay invisible while early results arrive quickly. A
permissive interface is one of the easiest ways to take that debt on.
Two statements frame everything that follows. First, ftlab is educational and
runs offline on a CPU. It loads no pretrained model and performs no production
operation. Second, an audit or a plan is not training. The commands that
inspect and freeze a configuration execute no model, acquire no data and grant no
permission. Every command and output in this chapter was run against the
installed wheel, ftlab version 0.12.0, in a fresh virtual environment outside
the source checkout. The output lines are reproduced verbatim apart from the
directory prefix.
Five states that must not collapse
A fine-tuning workflow passes through states that are easy to treat as one:
- Plan. Decide what will be run: the configuration, the source, the environment, the task, the budget and the stages.
- Authorise. Establish that the run is permitted: rights to the data, permission to train, and the resources it may use.
- Execute. Run exactly what was planned, and record what happened.
- Evaluate. Recompute the decision from the recorded observations, and check that the record is internally consistent.
- Accept. A person, with the authority to do so, decides that the result is adequate for its purpose.
Part VII, in “Operating, Auditing and Rolling Back a Fine-Tune”, kept review,
authorisation, activation and rollback separate for a release. The same
separation applies to a run. ftlab implements plan, execute and evaluate. It
does not implement authorise or accept, and it is designed so that it cannot be
mistaken for doing either. Authority comes from contracts and directives that
exist outside the configuration. Acceptance is a human act that no command
performs.
The installed command
The command is installed by standard console-script metadata, which maps a
command name to a Python function. The
setuptools entry-point documentation
describes the mechanism, and the
Python Packaging User Guide
describes the modern [project.scripts] form. FineTuneLab uses a slightly
unusual route. Part VII’s frozen plans record the hash of the package’s
pyproject.toml, so changing that file would invalidate accepted records. The
file was left byte-for-byte unchanged, and a small setup.py adds the entry
point ftlab = ftlab.cli:main during the build. The same function is reachable
as python -m ftlab.
Help lists six subcommands:
$ ftlab --help
usage: ftlab [-h] {audit,plan,run,demo,report,compare} ...
Offline educational FineTuneLab; no pretrained or production execution
positional arguments:
{audit,plan,run,demo,report,compare}
options:
-h, --help show this help message and exit
flowchart LR
A[audit: check contract] --> P[plan: freeze config + env + graph]
P --> R[run: refuse unless plan current]
R --> E[report: verify seals, recompute decision]
E --> C[compare: one task, one split]
P -. wrong backend .-> U[UNSUPPORTED_BACKEND, exit 3]
R -. stale plan .-> S[PLAN_CURRENT]
Every subcommand requires --output, and the output directory must not exist.
audit and plan take --config. run takes --config and --plan. demo
takes --profile, --seed and --steps. report takes --run. compare takes
--left and --right. There are no other options. A successful command prints
one JSON line naming the command, the output and the status
COMPLETED_EDUCATIONAL_ONLY. A failure prints a JSON object with an error code
and detail to standard error and exits with status 3 if the failure concerns
authorisation or an unsupported backend, and 2 otherwise. Two exceptions: a
malformed command line gets argparse’s usage message instead, and a missing file
puts the operating system’s message where the code would be.
The configuration
A configuration has exactly six fields, and every one is required:
{
"schema_version": "1.0",
"profile": "incident",
"seed": 37,
"steps": 8,
"output_id": "reader-incident-run",
"backend": "rule"
}
The profile names a task. There are five, and each is bound to one backend:
| Profile | Backend | What it does |
|---|---|---|
| incident | rule | Audits the canonical incident contract and evaluates the rule control |
| docket | constructed-linear | Runs the Part VII constructed docket system, export, loopback HTTP and rollback rehearsal |
| tiny-copy | tiny-transformer | Trains a random tiny model on an educational copy task, then adapts it |
| preferences | objective-arithmetic | Computes a preference objective on supplied numbers |
| recovery | resume-net | Interrupts and resumes a small classifier’s training |
The pairs are fixed. A configuration that pairs a profile with any other backend is refused. The seed must be 37 or 38, the two seeds declared for every experiment. The steps field must be between 1 and 16, and at least 2 for recovery, which needs a boundary at which to interrupt. The output_id must be a simple identifier of letters, digits, hyphens and underscores, at most 64 characters, starting with a letter or digit. Files larger than 32,768 bytes, duplicate keys and non-finite numbers are refused.
What the configuration cannot contain matters as much. There is no field for approval, for training permission, for a data path, for a model name or for a device. A user cannot turn a gate off by adding a key, because unknown keys are refused. Here are the refusals, as run:
$ ftlab audit --config bad-extra-approval.json --output bad-out-extra-approval
{"error": "CONFIG_FIELDS", "detail": "CONFIG_FIELDS: Exact configuration; approvals and training_allowed are not grant fields"}
$ ftlab audit --config bad-wrong-backend.json --output bad-out-wrong-backend
{"error": "UNSUPPORTED_BACKEND", "detail": "UNSUPPORTED_BACKEND: Profile/backend pair must match"}
$ ftlab audit --config bad-seed-39.json --output bad-out-seed-39
{"error": "SEED", "detail": "SEED: At most the two declared seeds"}
$ ftlab audit --config bad-path-id.json --output bad-out-path-id
{"error": "CONFIG_PATH", "detail": "CONFIG_PATH: Simple output identity, never paths or URLs"}
$ ftlab audit --config bad-recovery-1.json --output bad-out-recovery-1
{"error": "TRAIN_BUDGET", "detail": "TRAIN_BUDGET: Recovery requires an interruption boundary"}
$ ftlab audit --config bad-duplicate.json --output bad-out-duplicate
{"error": "CONFIG_JSON", "detail": "CONFIG_JSON: Duplicate JSON key: profile"}
The first file added a field called approved; a second test with
training_allowed gave the same refusal. The wrong-backend case exited with
status 3, the others with status 2. An unknown profile gave TASK_UNDECLARED,
and 17 steps gave TRAIN_BUDGET.
Contracts and rights
The incident profile carries an installed copy of the canonical incident-triage
task contract and its p1 policy, parsed by the existing ftlab.contracts module.
That contract is a draft, and it says training_allowed = false. Copying it into
the wheel does not change its status and does not freeze a benchmark. The
incident profile therefore cannot train, and no configuration can make it. The
audit records this directly:
"production_training": {
"status": "NOT_RUN_AUTHORIZATION",
"reason": "Incident contract draft, training_allowed=false; no benchmark freeze"
}
The tiny-copy and recovery profiles do train, on fixtures created for the book,
under an explicit author directive recorded in every plan as
author-directive:F-FP-P8-CODEX, with production_authority set to NONE. That
directive permits bounded CPU training of random tiny models on those fixtures and
nothing else. Rights questions, the subject of Chapter 13, “Dataset Rights,
Provenance and Privacy”, do not arise for these fixtures because every record was
generated by the book’s code. They would arise immediately for any real data, and
ftlab has no way to accept real data.
The typed graph
ftlab plan freezes a configuration into a plan with schema
p8-capstone-plan/1. The plan records the configuration, the task identity (the
hash of the task contract or fixture), the SHA-256 of every Python and JSON file in the installed package, computed
after normalising line endings, a digest over those hashes, the environment (Python and Torch
versions, platform, device and thread count), a budget, a training permission and
a typed graph. The plan for the incident configuration above has three nodes:
| Node | Operation | Input kind | Output kind | Permission |
|---|---|---|---|---|
| audit | audit | configuration | audit | READ_ONLY |
| execute | incident | audit | raw_observations | SYNTHETIC_MECHANISM |
| decide | report | raw_observations | decision | READ_ONLY |
The edges run from audit to execute, carrying an audit, and from execute to
decide, carrying raw observations. Each edge also carries the task identity, so a
stage cannot consume an artefact made for a different task. For the tiny-copy and
recovery profiles the execute node’s permission is
AUTHOR_DIRECTIVE_P8_BOUNDED_CPU instead. The budget declares the device as the CPU with one thread, each stage at most 180
seconds, the network off, the model origin as random tiny or constructed models,
and at most 2 seeds. It declares optimiser updates as zero for every profile
except tiny-copy, at three times the steps, and recovery, at twice the steps.
These limits are declared and only partly enforced. Seeds and steps are refused at
configuration time, and stage time is checked after a profile finishes and per
training step, but there is no external timer that stops a stage, no memory
quota, and the update count follows from the code rather than being checked
against the plan. The docket profile makes loopback HTTP requests, so the network
flag means no external network. The plan carries its own digest, a seal over
its contents.
The figure shows the three-node graph, what the frozen plan binds, and how receipts chain by artefact hashes. Authority stays outside the configuration in contracts and directives; blocked branches are listed, not run. It describes the implemented ftlab 0.12.0 workflow.
The execute node is deliberately coarse. Inside it, each profile runs its own stages: a baseline, training and evaluation for tiny-copy; export, loopback HTTP and rollback for docket. They are not flattened into one generic pipeline, because they are not interchangeable. A tiny transformer’s adapter cannot be exported through the docket’s export path, and the result says so.
Identities and receipts
ftlab run executes only if the plan is current. It recomputes the plan from the
configuration, the installed source and the live environment, and refuses with
PLAN_CURRENT if anything differs. Changing one value in the configuration,
steps from 8 to 4, was enough:
$ ftlab run --config incident-changed.json --plan incident-plan/plan.json --output incident-stale
{"error": "PLAN_CURRENT", "detail": "PLAN_CURRENT: Stale source, environment, task, tokenizer/backend, permission or configuration"}
The same check means that a plan frozen on one machine cannot be run on a machine with a different Torch build. The plans in the book’s evidence record a CPU-only Torch build; the verification environment for this chapter had a different build of the same version, so its plans differed, and it could not replay theirs. That is the intended behaviour. A changed environment needs a new plan, not an edited receipt.
A run writes result.json, with schema p8-capstone-result/1. It contains the
audit, the raw observations, the decision and three stage receipts. Each receipt
names its node, the task and the source digest, and records an input identity
and an output identity. The audit receipt’s input is the plan’s digest. Each
later receipt’s input is the previous receipt’s output, and each output is the
hash of the artefact that stage actually produced. The receipts are sealed
individually, and the result as a whole is sealed. The result also lists the
branches that were not run and why:
[{"operation": "pretrained/real-training/gpu/production-release", "status": "NOT_RUN_AUTHORIZATION"},
{"operation": "tiny-transformer-to-docket-export", "status": "UNSUPPORTED_BACKEND"}]
ftlab report checks every seal, confirms that the result belongs to the plan,
validates the graph, confirms that the three receipts are connected and that
their output identities equal the hashes of the recorded audit, observations and
decision, recomputes the decision from the observations, and confirms that the
plan is still current. Only then does it write report.txt and decision.json.
A seal is a hash. It detects accidental change and inconsistent records. It is
not a signature, and it does not prove who ran the command or on what machine.
Every result says so in the field authenticated_execution, whose value is
NOT_ESTABLISHED.
Decision records
The decision is computed by a fixed function of the profile and, for tiny-copy
only, the raw observations. It is not an opinion the code forms, and it is not a recommendation
to deploy. For the incident profile the decision is always DO_NOT_TRAIN,
with training_success_claim set to false; the code returns it for the profile,
and the reason it records is that the contract forbids training. For docket the
decision is always KEEP_RULE_BASELINE, with the three systems’ task scores
attached; the code does not compare them, and the recorded scores are what
justify the fixed outcome. For
tiny-copy it compares exact-match success on the heldout rotation task and
reports retention and export status. For preferences and recovery it returns
COLLECT_MORE_EVIDENCE, because those profiles exercise a mechanism on a
different task and say nothing about which model is better. Chapter 47, “FineTuneLab:
Run and Compare the Training Pipeline”, reads each decision in detail.
The end-to-end acceptance run
Record FT-P8-INTEGRATED-02-P8-46A-s37 ran the four explicit commands, audit, plan,
run and report, as four separate subprocesses through the real entry points. All
four exited with status 0. The incident fixture has five synthetic cases. The rule
control returned the exact gold output on all five, covered four of them and
abstained on the fifth, whose information status is insufficient, so coverage is
0.8. Two cases have high severity, and neither was missed. Training was
NOT_RUN_AUTHORIZATION. Approval status was APPROVAL_VERIFICATION_PENDING. The
decision was DO_NOT_TRAIN.
The 5/5 needs an honest reading. The gold labels and the rule’s predictions are both computed from the same routing policy. The rule is, in effect, the oracle. Agreement is guaranteed by construction and is not evidence that the rule would generalise to real incidents. The value of the run is elsewhere: it is a complete workflow, from configuration to decision, that ends in a well-justified refusal to train, and that refusal is the correct outcome for a task whose contract does not permit training.
The same four commands, repeated for this chapter in a separate environment, produced the same decision. The report reads:
FineTuneLab educational report
{
"recommendation": "DO_NOT_TRAIN",
"reason": "Training forbidden by canonical draft contract; useful rule-control report remains available",
"training_success_claim": false
}
Authenticated execution: NOT_ESTABLISHED
See result.json for raw observations and blocked branches.
Running the report command a second time into the same directory was refused
with OUTPUT_EXISTS. No output is ever overwritten.
Supported and unsupported operations
ftlab supports: auditing and planning the five profiles; running them on a CPU
within the budget; the demo shortcut, which writes a configuration and plan and
then runs; verifying and reporting a run; and comparing two token-evaluation
envelopes that share task, data, evaluator, split and gold.
It does not support, and refuses rather than approximates: any pretrained or user-supplied model, any user-supplied dataset, any GPU, any network access, any training on the incident contract, any export of a tiny-transformer adapter through the docket path, any production deployment, and any comparison across tasks or with training-split data. There is no generic checkpoint loader. No CPU fallback silently replaces a requested GPU operation, because there is no GPU operation to request.
Negative controls
Beyond the refusals shown above, the package tests refuse forged approval fields, path-traversal identifiers, unsupported backends, drift in any source file, reused output directories and plans whose typed edges have been swapped. The comparison tests refuse training splits, unpaired identifiers, mismatched gold and exact-match flags that disagree with the generated tokens. The installed-wheel test builds the wheel offline, installs it into a separate virtual environment, removes checkout paths from the import path, changes directory, and confirms that every imported FineTuneLab module comes from the installed environment.
These checks establish tested local boundaries. They cannot establish how a caller obtained the data inside a comparison envelope, or that a claimed holdout really was held out. Provenance outside the receipts remains the caller’s responsibility.
Limitations
The integrated wheel declares jsonschema==4.26.0 as a mandatory runtime
requirement; the isolated dependency gate resolved it without global packages.
It does not bundle Torch or the numerical environment. It was verified in the book’s pinned
container environment and in one separate Linux environment with Torch already
installed. That is a fresh FineTuneLab installation, not a demonstration of
installation on an empty machine. Native Windows is not a verified runtime. The
version, 0.12.0, is unchanged; choosing a release version is part of
integration, not of this chapter.
Practical consequences
- Make the configuration exact and refuse unknown fields, so that no gate can be turned off from the front door.
- Fix the pairing of task and backend, and refuse every other pairing.
- Freeze plans against source, environment and configuration, and require a new plan for any change.
- Chain stage receipts by the hashes of the artefacts each stage produced.
- Keep authorisation in contracts and directives that the configuration cannot edit, and leave acceptance to a person.
- State in every record whether execution is authenticated. Here it is not.
FineTuneLab
ftlab.cli implements the six subcommands. ftlab.capstone.config reads and
validates configurations, defines the profiles, builds plans, validates graphs and
checks currency. ftlab.capstone.workflow audits, executes profiles, seals
receipts, verifies results, recomputes decisions and checks comparability. They
depend on the existing contracts, delivery, learning and advanced modules, and add
no training method of their own.
Where this leads
The interface is strict, but it is only an interface. The next chapter, “FineTuneLab: Run and Compare the Training Pipeline”, runs what lies behind it: an actual, small training run with its loss history, checkpoint and evaluation, set beside a no-training baseline, a preference objective and a recovery test, without ever merging their results into one claim. It also asks the question the whole book has been building towards: when should a model be fine-tuned at all?
References
- Python Packaging Authority. Writing your pyproject.toml: creating executable scripts. Python Packaging User Guide.
- Setuptools developers. Entry Points. Setuptools documentation.
- Sculley, D., Holt, G., Golovin, D., Davydov, E., Phillips, T., Ebner, D., Chaudhary, V., Young, M., Crespo, J.-F. and Dennison, D. (2015). Hidden Technical Debt in Machine Learning Systems. Advances in Neural Information Processing Systems 28.
Further Reading
- Daniel Sculley et al., “Hidden Technical Debt in Machine Learning Systems” (2015). Read the source. Why strict interfaces matter: the debt taxonomy the exact configuration is built to resist.
- Joelle Pineau et al., “Improving Reproducibility in Machine Learning Research” (2021). Read the source. Evidence on reproducibility practice behind frozen plans and sealed receipts.
- Setuptools developers, “Entry Points” (documentation). Read the source. The console-script mechanism behind the installed
ftlabcommand.