← Fine Tuning From First Principles

FineTuneLab: Run and Compare the Training Pipeline

Run FineTuneLab's pipeline branch by branch: a contract that forbids training, a constructed docket system beside its rule control, a real tiny training run whose loss falls while every output stays constant, a preference objective computed by hand and an interrupted run that resumes exactly. Then decide, from that evidence, when fine-tuning is justified.

The first chapter of this book, “What Fine-Tuning Actually Changes”, began with a parameter update. The second, “When Fine-Tuning Beats Prompting, RAG, Tools or Distillation”, asked when such an update is worth making. Forty-four chapters later, FineTuneLab has every piece needed to answer that question with evidence rather than habit: contracts, baselines, masked losses, adapters, checkpoints, evaluators, statistics and a strict interface. This chapter runs them.

Its central question is the book’s: when should a model be fine-tuned? The chapter answers it by running five branches of the pipeline, reading each against its own baseline, and refusing to combine them into a single story. One branch trains a real model. Its held-out loss falls under every trained condition, its training loss falls in most of its stages, and it does not get a single additional answer right. That result is the most useful thing in the chapter.

The branches are different experiments

The interface described in “FineTuneLab: Design the Usable Solution” exposes five profiles. Each is a different task with different data, a different model (or none), a different objective and a different evaluator. They share commands and receipts. They do not share a denominator, a model or a training grant.

Branch Task Data Model Status Objective Steps Evaluator Outcome
incident Canonical incident triage 5 synthetic cases Rule control Training NOT_RUN_AUTHORIZATION None 0 Exact structured output DO_NOT_TRAIN
docket Part VII docket 24 synthetic test cases Base, constructed adapter, rule Constructed, not trained None 0 Task success KEEP_RULE_BASELINE
tiny-copy Copy to rotation 24/8/16 per task Random tiny transformer + LoRA Trained Masked next-token NLL 8 + 8 + 8 Exact match, NLL KEEP_BASELINE
preferences Preference objective 2 supplied pairs None Arithmetic; fitting NOT_RUN_OPTIONAL DPO-style loss 0 Loss value COLLECT_MORE_EVIDENCE
recovery Binary features Synthetic binary features ResumeNet with LoRA head Trained Cross-entropy 8 State equality COLLECT_MORE_EVIDENCE

Loss histories and checkpoints also differ by branch. The incident, docket and preference branches have no loss history and no training checkpoint; the docket branch’s v1 and v2 exports are release artefacts, not training state. Tiny-copy has an eight-step history for each of its three stages and a saved and reloaded adapter. Recovery has an eight-step history and a full training-state checkpoint at step 4. Both are given below.

Two branches train. Neither trains on the incident or docket task, and no result from one branch can be added to, averaged with or compared against another. A statement such as “the pipeline improved the system” would have to choose one branch and one denominator, and the honest version of that statement, for every branch, is given below.

Five pipeline branches with own baselines and terminal decisions

The figure keeps the five denominators apart and names each terminal outcome. The tiny-copy panel recalls the constant-output finding: held-out NLL falls under every trained condition while exact match never moves. Unexecuted milestones stay greyed as not run; nothing here implies they ran.

The evidence comes from the FT-P8-INTEGRATED-02 namespace, which was planned and frozen before execution. Twelve records were produced from nine executed proposals. Two proposals with real models or real data, P8-47D and P8-48B, and the real long-context and multimodal probe, P8-45C, are NOT_RUN_AUTHORIZATION. The optional learned branches P8-43B, P8-44B and the preference fitting in P8-47B are NOT_RUN_OPTIONAL. All training used a CPU with one thread.

Branch one: the contract that forbids training

The incident branch is the shortest and the most important. The canonical incident-triage contract is a draft, and it sets training_allowed to false. The audit stage reads it and records NOT_RUN_AUTHORIZATION for training. The run evaluates the rule control on five synthetic cases: four answered and one abstention on a case whose information is insufficient, all five matching the gold output exactly, no high-severity case missed. The decision is DO_NOT_TRAIN, with training_success_claim set to false. Record FT-P8-INTEGRATED-02-P8-46A-s37 holds the evidence.

The 5/5 is not evidence that the rule is good. Gold and prediction are computed from the same routing policy, so they agree by construction. What the branch demonstrates is that a well-built pipeline can reach a correct and useful decision, not to train, without running any training at all.

Branch two: the baseline that defeats the adapter

The docket branch, record FT-P8-INTEGRATED-02-P8-48A-s37, evaluates three systems on the same 24 test cases of the Part VII docket task:

System Task successes Rate
Base constructed system 8/24 0.333
Constructed adapter 24/24 1.000
Rule control 24/24 1.000

The base system answers “complete” to every request and is right exactly on the 8 cases whose gold action is complete. The constructed adapter, whose weights were written down rather than learned, gets all 24 right. So does the rule, which applies the policy directly. The decision is KEEP_RULE_BASELINE. Its recorded reason is that an oracle-sharing control that needs no training matches the constructed adapter on this finite fixture. When a few lines of policy code achieve the same result as an adapter, the adapter adds a model to maintain, an export to verify and a release to operate, and buys nothing. Chapter 6, “A Baseline Can Defeat the Fine-Tune”, made this argument in general. Here it is the recorded outcome.

Branch three: a real training run

The tiny-copy branch is the only branch that trains a language model. Records FT-P8-INTEGRATED-02-P8-47A-s37 and FT-P8-INTEGRATED-02-P8-47A-s38 hold it.

Task and data. Every prompt is a letter from a to h followed by a digit, such as a4. The source task copies the letter, so a4 → a. The adaptation task returns the cyclic successor, so a4 → b and h4 → a. For each task, digits 0 to 2 give 24 training cases, digit 3 gives 8 validation cases and digits 4 and 5 give 16 held-out test cases. Each answer is one letter and an end-of-sequence token, so 16 test cases have 32 supervised targets.

Model. A randomly initialised tiny transformer: one layer, width 8, 2 heads, hidden size 16, context 16, a 262-token byte vocabulary and no dropout. It is not pretrained and has no prior knowledge of letters.

Stages. First, the source stage trains the full model for 8 updates on the copy task, with AdamW at learning rate 0.01, batch size 4, weight decay 0 and gradient clipping at 1.0. The result is the baseline for everything that follows, so the comparison asks whether adaptation improves on a source-trained model, not on random weights. From the baseline, two branches run. Source continuation trains all parameters for 8 more updates on the copy task, at the same learning rate. Adaptation freezes the baseline and trains a LoRA adapter of rank 2 and alpha 2 on the attention projection blocks.0.project for 8 updates on the rotation task, at learning rate 0.02. LoRA is the low-rank method derived in Chapter 22, “LoRA From Linear Algebra”. The adapter is saved, reloaded and evaluated again. That is 24 optimiser updates per seed and 48 across both seeds.

Objective. The loss is the masked next-token negative log-likelihood of the assistant answer:

$$ L(\theta) = -\frac{1}{|\mathcal{A}|} \sum_{i \in \mathcal{A}} \log p_\theta(x_i \mid x_{\lt i}), $$
where 𝒜 is the set of supervised answer positions. Prompt tokens and padding receive label −100 and contribute nothing. For reference, a model that assigns uniform probability over 262 tokens has an NLL of ln 262 ≈ 5.568.

Evaluation. Greedy generation of at most two new tokens. A case is an exact match only if the generated tokens are exactly the answer letter followed by the end token. Mean NLL is reported over the 32 test targets.

Separation. Every batch in all 48 updates draws its identifiers from a training split. The source and continuation stages see only copy-task training cases. The adaptation stage sees only rotation training cases. Validation and test cases are used only for predeclared reporting. Hyperparameters were fixed before the run and never selected on validation or test data. The authoring cross-check confirmed the split separation from the batch identifiers in every recorded step.

The loss histories

The per-step training losses for all eight updates of each stage, rounded to four decimals, were:

Seed Stage 1 2 3 4 5 6 7 8
37 Source 5.5942 5.2893 5.2011 5.5615 5.6259 5.2014 4.5910 4.3310
37 Continuation 4.3581 4.0568 4.1497 4.4818 4.5093 4.3216 3.4611 3.1369
37 Adaptation 4.1849 4.2885 4.4921 5.1906 4.6213 4.5466 4.1806 4.2847
38 Source 6.0320 5.6728 4.9766 5.0911 4.9044 4.5811 4.8996 4.7700
38 Continuation 4.5641 4.4927 4.2415 4.5391 3.9157 3.8176 3.5993 3.6730
38 Adaptation 4.7464 4.3738 4.7165 4.1358 4.1853 4.3684 4.7467 4.3738

These are batch losses on different batches, so step-to-step movement is noisy; the seed 37 adaptation loss ends higher than it starts. The authoring cross-check recomputed every one of the 24 step losses at each seed from the logits recorded before each update, using a log-sum-exp in the standard library, and matched the recorded values.

The results

On the 16 held-out rotation cases:

Seed Condition Exact match Mean NLL Source retention
37 Baseline 0/16 4.564 0/16
37 Source continuation 0/16 3.421 0/16
37 Adapted 0/16 4.559 0/16
37 Reloaded adapter 0/16 4.559 0/16
38 Baseline 2/16 4.422 2/16
38 Source continuation 2/16 3.482 2/16
38 Adapted 2/16 4.421 2/16
38 Reloaded adapter 2/16 4.421 2/16

The reloaded adapter has the same identity and the same results as the adapter before saving, so the save and reload preserved the parameters. The paired exact-match effect of adaptation over the baseline is 0 out of 16 at both seeds. The decision is KEEP_BASELINE. Export is UNSUPPORTED_BACKEND, because the tiny transformer’s adapter has no route into the docket export path; serving it remains blocked.

What the outputs actually were

A table of rates hides the most important fact, which the authoring cross-check found by reading every generated sequence. Every condition, at both seeds, produces one constant output for all 16 test prompts. At seed 37, the baseline and adapted models always emit byte 162, which is not a letter, followed by the end token; the continued model emits only the end token. At seed 38, the baseline and adapted models always emit g, and the continued model always emits a.

That explains seed 38’s 2/16. The rotation answer is g for the prompts f4 and f5, so a model that always says g is right twice. The source retention of 2/16 has the same cause: g4 and g5 copy to g. The continued model’s a is right for h4 and h5 under rotation and for a4 and a5 under copying. None of these models has learned either task. The 2/16 is a constant guess that happens to coincide with two gold answers, and the 0/16 at seed 37 is the same behaviour with a guess that coincides with nothing.

Now look at the loss. Source continuation, which never saw a rotation example, reduced rotation test NLL from 4.564 to 3.421 at seed 37. The rotation adapter, trained specifically on the rotation task, reduced it from 4.564 to 4.559. By the loss, the wrong training was about 240 times more effective than the right training at seed 37, and about 590 times at seed 38. Much of this reduction comes from the 16 end-token targets among the 32 scored positions. Independent decomposition of the raw logits shows that both the end-token loss and the letter-token loss fall during source continuation; the end-token term accounts for most of the total reduction. Thus the reduction is not exclusively an EOS effect. Neither improvement produces prompt-dependent letter predictions: the model still emits the same answer on every prompt. Learning the shared output format, or changing confidence without changing the answer, can lower NLL without establishing the task behaviour.

This is the clearest demonstration in the book that lower loss is not utility. A lower held-out loss can come from the easy, shared part of the output while the part that matters is untouched. A decision rule that selected the candidate with the lowest validation loss would have chosen source continuation, which at seed 37 answers every question with an empty string and at seed 38 answers every question with a.

Branch four: a preference objective by hand

Record FT-P8-INTEGRATED-02-P8-47B-s37 computes the Direct Preference Optimization loss, derived in Chapter 32, “Direct Preference Optimization From First Principles”, on two supplied pairs. With β = 0.2, policy log-probabilities for the chosen and rejected answers, and reference log-probabilities for the same,

$$ \ell_i = -\log \sigma\!\left(\beta \left[ (\log \pi_\theta(y_w) - \log \pi_\theta(y_l)) - (\log \pi_{\text{ref}}(y_w) - \log \pi_{\text{ref}}(y_l)) \right]\right). $$
The first pair has policy log-probabilities −1 and −2 and reference values −1.5 and −1.5, so the margin is 0.2 × (1 − 0) = 0.2 and the loss is ln(1 + e^(−0.2)) = 0.598139. The second has policy values −2 and −1, a margin of −0.2 and a loss of 0.798139. The mean is 0.698139. No model was fitted, no preference data was used and no holdout was consumed. The decision, `COLLECT_MORE_EVIDENCE`, says only that an objective was computed on a different task and ranks no model.

Branch five: interruption and recovery

Records FT-P8-INTEGRATED-02-P8-47C-s37 and FT-P8-INTEGRATED-02-P8-47C-s38 reuse the recovery task from Part V. A small classifier with 4 inputs, 8 hidden units, 2 outputs and dropout 0.25 has a rank-2 LoRA adapter on its output head. It is trained for 8 steps at a decaying learning rate starting at 0.02, with microbatches of 2 and gradient accumulation over 2, in float32. One run is uninterrupted. The other stops after step 4, saves its full training state, reloads it and finishes the remaining 4 steps.

At both seeds, the resumed run’s per-step history, including losses, learning rates, batch indices, predictions and dropout noise, equals the uninterrupted run’s, and the final state digests are identical. The per-step losses of the uninterrupted run, identical in the resumed run, were 0.7209, 0.7266, 0.8101, 0.7912, 0.7356, 0.6799, 0.6711 and 0.6436 at seed 37, and 0.5957, 0.8616, 0.6950, 0.8293, 0.7547, 0.5548, 0.7827 and 0.7566 at seed 38. The checkpoint written at step 4 holds the full training state in 21,479 bytes at seed 37 and 21,489 bytes at seed 38. Chapter 30, “Checkpoints, Reproducibility and Recovery”, explained why exact resumption needs the optimiser state, the random number generators and the data order as well as the weights. This branch shows it holding on a CPU with one thread. It does not show that a run resumes exactly on a different machine or library build; PyTorch’s own reproducibility notes say that completely reproducible results are not guaranteed across releases or platforms, or between CPU and GPU.

Comparability and statistics

Only one branch, tiny-copy, compares trained candidates on one task, and the comparison is fully paired: the same 16 prompts, the same gold answers, the same evaluator and the same split. The compare command enforces exactly that. When envelopes holding the baseline and adapted predictions from a 2-step tiny-copy demo were compared, it reported:

{"left": 0, "right": 0, "denominator": 16, "effect": 0.0}

Replacing the right-hand split with train, or its evaluator with a different one, was refused with INCOMPARABLE_TASK. Setting one exact-match flag to true without changing the generated tokens was refused with TEST_ACCESS.

With an effect of zero and constant outputs, no confidence interval is informative. Chapter 38, “Statistical Confidence and Ablation”, showed how to resample paired cases. Resampling sixteen identical differences of zero yields zero every time. Two seeds is all the plan allowed, and seed variation in fine-tuning is known to be large: Dodge and colleagues found that changing only the random seed when fine-tuning a pretrained model produced large performance swings. Here the seeds differ in a constant guess, which is a reminder that a difference between seeds can be an artefact as easily as a signal.

Cost and retention

Each tiny-copy seed took 24 optimiser updates. The recorded time of the training function was 1.044 seconds at seed 37 and 0.201 seconds at seed 38, for one CPU thread; the whole records, including evaluation and receipts, took 1.716 and 0.719 seconds. A single run’s timing probably includes warm-up and says nothing about larger models. Retention was measured as exact match on the held-out copy task after each condition. It did not change from the baseline in any condition, 0/16 at seed 37 and 2/16 at seed 38, and since both are constant outputs, the honest summary is that there was no source skill to retain.

Do not fine-tune: a worked decision

Put the evidence together for a team that wants to fine-tune the incident-triage system, and walk through the decision in order.

Is training permitted? The task contract is a draft with training_allowed false. The answer is no, and the process could stop here. The remaining steps show that stopping is also the right answer on the merits.

Is there a baseline, and how good is it? The rule control agrees with gold on 5/5 incident cases, but gold and rule share their policy, so that agreement says nothing about the rule’s quality on real incidents, which is unmeasured. By analogy, the related docket task shows the pattern to expect when a rule encodes the policy: the rule is correct on 24/24 and the constructed adapter, the best any adapter could do on that fixture, only ties it.

Is the remaining error something training can fix? No residual error has been measured, because there is no evaluation of the rule independent of the policy it implements. The first piece of work is therefore an evaluation set with gold labels that do not come from the rule, not a training run. Until it shows an error that the rule cannot express, there is nothing for a model to learn.

Would a training run be evaluated on the task? The only executed training run, on a different task, shows by analogy what to watch for: held-out loss fell under every trained condition, and exact match did not move. A proposal that reports training loss, or even held-out loss, as its evidence of improvement would, at seed 37, have chosen a model that outputs nothing.

Can the result be served and rolled back? There is no serving path for a trained incident model in FineTuneLab. The only trained adapter it can produce, the tiny transformer’s, has no export path (UNSUPPORTED_BACKEND), and the only release path it rehearses is for a constructed docket system. A trained incident model would need its own export, serving and rollback before release could even be considered.

What would it cost? A model to maintain, an export and identity check to verify, a release to authorise and operate, and a rollback to rehearse, all to tie a rule that costs none of those.

The recommendation is not to fine-tune. Keep the rule, write down the conditions that would reopen the question (a real error the rule cannot express, a task contract that permits training, rights to representative data, and a fixed evaluation set), and revisit when one of them changes. This is the same reasoning Chapter 2 began with. Retrieval, as in retrieval-augmented generation, a better prompt, a tool or a rule should each be tried and measured before an update to the weights.

When fine-tuning is justified

The branches also show, by contrast, what a justified fine-tune would need. Each condition below has appeared somewhere in the book; here they form the decision criteria.

  1. Permission. The task contract allows training, and the data rights are established.
  2. A measured gap. The strongest non-training baseline, including prompting, retrieval, tools and rules, has been measured on the task and leaves an error that matters.
  3. A learnable gap. The error is in the model’s behaviour, not in the surrounding system’s permissions, tools or post-processing.
  4. A fixed, separated evaluation. Validation and test sets were fixed before training, with families isolated, and are used only for predeclared purposes.
  5. Task-level success. The candidate improves the task metric, not only the loss, on paired cases, by more than a declared practical threshold, with uncertainty reported.
  6. No unacceptable regression. Slices and retained skills are reported against named comparators, as in “Multilingual and Multi-Task Adaptation”.
  7. A release path. The adapter can be exported, served, identified, monitored and rolled back.
  8. A cost case. The improvement is worth the model, the operations and the evidence it will require.

FineTuneLab encodes part of this, and only part. Its decision function for tiny-copy returns COLLECT_MORE_EVIDENCE if the paired exact-match effect is positive and KEEP_BASELINE otherwise. Even a positive effect does not produce a recommendation to deploy. Criteria 1, 4 and 7 are only partly enforced: training permission is enforced by the contract and the training permit, but there is no data-rights check, only the absence of any way to supply real data; the tiny-copy training stage checks that every batch comes from a training split, but it does not check family isolation; and the export check only records UNSUPPORTED_BACKEND for tiny-copy rather than providing serving, monitoring or rollback. Criteria 2, 3, 6 and 8 are judgements this chapter describes and the code does not make.

Negative controls

The tiny-copy branch is its own negative control for the loss metric: source continuation is a condition that should not help rotation, and it lowered rotation loss the most. The reload check confirms that adapter identity survives saving. The comparison refusals above confirm that train-split and cross-evaluator comparisons cannot be made. The authoring cross-check recomputed every test and validation NLL from the 262 recorded logits per target, every exact-match count from the generated tokens, every step loss, the preference losses and the recovery equalities, without importing FineTuneLab.

What the chapter shows and does not show

It shows a working pipeline that trains, checkpoints, reloads, evaluates, compares and decides, and that reaches DO_NOT_TRAIN, KEEP_RULE_BASELINE and KEEP_BASELINE for reasons it records. It shows a real case in which loss falls and utility does not. It shows exact resumption of an interrupted run.

It does not show that fine-tuning works on any real task. No pretrained model was loaded, no real data was used, no GPU was involved and no QLoRA or preference training was run in this part. The tiny transformer learned neither of its tasks. The book’s acceptance criteria call for a LoRA run on a pinned open model and a fixed benchmark comparing prompting, retrieval and training. Those remain NOT_RUN_AUTHORIZATION and are a blocking limitation of the book’s evidence, not a completed result.

Limitations

The trained model is tiny, random and evaluated on 16 cases per task with two seeds. The incident and docket results share their oracle with the rule. The preference branch is arithmetic. The recovery branch is a separate classifier. Timings are single-run CPU measurements. Every decision is educational and has no production authority.

Practical consequences

  • Report every branch with its own task, data, model, status, objective, steps, evaluator and outcome, and never pool them.
  • Fix the baseline before training, and make it the strongest non-training system available.
  • Read the generated outputs, not only the rates. A constant output can produce a plausible-looking score.
  • Never select a model on loss alone. Lower held-out loss can come entirely from shared formatting tokens.
  • Keep validation and test out of every training batch, and check it from the batch identifiers.
  • Treat seed differences with suspicion until the outputs explain them.
  • Write down the conditions under which a refusal to fine-tune would be revisited.

FineTuneLab

ftlab.capstone.training runs the tiny-copy source, continuation and adaptation stages using the earlier learning and adapter modules. ftlab.capstone.workflow runs the docket, preference and recovery profiles through the Part VII delivery, Part VI objectives and Part V recovery modules, and computes every decision. The installed command ftlab demo --profile tiny-copy runs the training branch with a fresh output directory; the default steps are 8.

Where this leads

Every branch has now ended with a decision, and none of them is to release a trained model. The final chapter, “FineTuneLab: Release, Runbook and Decision Handbook”, takes the reader through the whole journey as a user would, from installation to a rehearsed rollback, and collects the book’s decisions into a handbook for the next dataset.

References

Further Reading

  1. Tim Dettmers et al., “QLoRA: Efficient Finetuning of Quantized LLMs” (2023). Read the source. The efficient-adapter method the book’s unexecuted open-model LoRA milestone would belong to; the milestone itself remains not run. Also recommended in Chapters 2 and 23–24.
  2. Long Ouyang et al., “Training Language Models to Follow Instructions with Human Feedback” (2022). Read the source. The credible train-and-improve case that the chapter’s refusal to fine-tune must be weighed against. Also recommended in Chapters 16 and 34.
  3. Jesse Dodge et al., “Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping” (2020). Read the source. Seed-variance evidence behind the warning to distrust differences between two seeds.