← Fine Tuning From First Principles

Supervised Fine-Tuning: A Verified Adapter Path

Connect supervised-token masks, frozen parameters, optimiser updates and adapter identities in a minimal training path.

A training loop finishes, the loss declines and an adapter file appears. None of those observations proves that the intended answer tokens were trained, that the base stayed frozen, or that the adapter can be loaded onto the intended checkpoint. The central engineering problem is to preserve those relationships through an actual update.

Supervised fine-tuning adapts a model using desired continuations. It does not require a new language-model loss: the causal objective from Chapter 08 remains. What changes is the data distribution, selected target positions and parameters allowed to move. A framework can hide those choices behind defaults; it cannot remove their consequences.

Our real pretrained-model experiment is pending authorisation. The executed reference uses a randomly initialised tiny CPU Transformer and independent character-pair fixtures. It exercises the training and adapter boundaries without using incident data, downloading weights or pretending that a toy loss decrease demonstrates Qwen adaptation.

Start from the experiment, not the trainer call

A genuine adapter study begins with an immutable checkpoint and tokenizer, permitted data access and a comparison protocol. The training plan must specify which examples may update parameters, which validation information may influence selection and which evaluation observations remain protected.

For the incident study, rights and benchmark authority are still pending. The 0.2.1 card makes the data’s measurement limits explicit; it does not grant use. Its existence is therefore insufficient to launch the following loop on its records.

The future bounded plan proposes rank-eight query/value adapters, sequence length 256, microbatch one, accumulation four and at most eight updates. It still needs actual module validation, precision placement, activation evidence and explicit author approval. These numbers are proposed workload limits, not a report of a completed experiment.

For the CPU mechanism, the model is far smaller: one block, width eight and a 16-position context. Only one output projection receives a rank-two adapter. The fixtures map a to b and aa to bb. Those associations have no connection to the incident label ontology or protected data.

Preserve the one causal shift

Let x contain the token sequence and y the unshifted labels. Ignored positions have label -100. With the selected target set J, the objective is

$$ \mathcal L(\theta)= -\frac{1}{|J|} \sum_{j\in J}\log p_\theta(x_j\mid x_{ The model logit at j-1 predicts the label at j. Our existing causal_terms function owns that single shift. The collator does not shift first, and the training loop does not shift again. Double shifting changes the task while leaving tensor shapes plausible.

The reference fixture obtains role-based loss masks from Chapter 16’s compile_supervision and uses Chapter 07’s existing batch constructor for padding. It does not add a parallel mask policy. The first user message conditions the answer but does not become an answer target. Assistant content and its selected EOS are supervised; padding and role markers retain their established treatment.

The two fixtures contribute two and three selected targets respectively: one or two answer bytes plus EOS. There are five targets per update. Their differing lengths make them a useful accumulation test, rather than merely repeating equal examples whose weighting errors cancel.

The byte vocabulary belongs to the educational model. Its IDs are not Qwen IDs, and its role template is not a verified native training template for a pretrained checkpoint. The real path must repeat the same conceptual checks with that checkpoint’s actual tokenizer and boundaries.

Freeze parameters without severing the computation

The adapter modifies a linear map while retaining the original matrix. In row-batch notation, the layer computes

$$ Y=XW^\top+\frac{\alpha}{r}(X\mathbf{A}^\top)\mathbf{B}^\top. $$
Chapter 22 derives the dimensions and gradients. Here we use the consequence: the optimiser should contain A and B, while W remains unchanged.

Setting W.requires_grad to false prevents accumulation of its parameter gradient. It does not mean evaluating the entire frozen base under no_grad. A loss gradient may still need to pass through a frozen transformation to reach an adapter in an earlier layer. Disconnecting that graph can make a correctly attached adapter untrainable.

The attachment function validates exact target-module names before mutating the model. It then freezes the model and inserts adapters at those targets. A missing target fails explicitly. A substring search that accidentally selects a different projection would otherwise change parameter count and experimental meaning.

In the width-eight reference projection, both input and output widths are eight. Rank two therefore introduces

$$ r(d_{\mathrm{in}}+d_{\mathrm{out}})=2(8+8)=32 $$
trainable parameters. The selected parameter list contains only the two factor matrices. Embeddings, attention, normalisation, MLP and base projection parameters remain frozen.

The harness does not merely trust a configuration label. Before the first update it derives the adapter set from module ownership, requires the trainable names to equal exactly *.A and *.B, and refuses TRAINABLE_SELECTION otherwise. It snapshots every non-adapter parameter, optimises only the adapter set and, after training, compares every non-adapter tensor with that snapshot. A changed base raises BASE_CHANGED. Saving and loading derive keys from LoRALinear ownership: a base tensor injected into adapter state fails with ADAPTER_KEYS even if its name ends in .A or .B, and a re-digested manifest cannot make the extra key acceptable.

This narrow target is chosen for inspection, not because it is an optimal adaptation strategy. A real checkpoint’s fused projections, storage convention and module names require direct verification.

Accumulate targets, not averages of unequal averages

Suppose microbatch k contributes a summed loss S_k over N_k selected tokens. The desired update uses

$$ \mathcal L_{\mathrm{update}}= \frac{\sum_k S_k}{\sum_k N_k}, \qquad \nabla\mathcal L_{\mathrm{update}}= \sum_k\frac{\nabla S_k}{N_{\mathrm{update}}}. $$
Backpropagating each microbatch's token sum divided by the common update count gives this gradient without retaining every graph simultaneously.

Averaging the two fixture means equally would give the two-target example half the update weight and the three-target example the other half. The token mean instead gives them weights 2/5 and 3/5. Neither is intrinsically the only possible learning objective, but they are different objectives. The implementation and experiment must agree on which one is intended.

The reference compares microbatch one with microbatch two under the same token-weighted objective. Their histories agree within the declared FP32 tolerance. Small reduction-order differences are not a licence to accept arbitrarily different gradients.

Effective sequence batch size is microbatch times accumulation count for this single-device plan. Effective selected-token count also depends on response lengths and masks. Reporting only “batch size four” leaves both the memory question and the loss denominator incomplete.

SFT: from selected masks to a reloadable adapter

The pipeline below carries selected targets through token-weighted accumulation into adapter-only updates while the base stays hashed and unchanged. Notice that reloaded logits agree exactly on the toy, which establishes the dependency chain without establishing any pretrained-model claim.

Make the optimiser state observable

The CPU reference uses AdamW with learning rate 0.02, zero weight decay and a gradient-norm ceiling of one. These choices define a bounded mechanism exercise; they are not suggested pretrained-model hyperparameters.

At each update the loop clears gradients, computes microbatch token sums, divides by the common target count, backpropagates, checks finiteness, clips the trainable gradient norm and steps the optimiser. It then measures the current fixture loss. No evaluation example influences an update or a selection decision.

Six CPU updates produce losses from approximately 5.54146 after the first update to 5.52822 after the sixth. These are post-update training losses on the same two fixtures. They do not establish held-out accuracy, fluency or superiority over a control.

For 32 FP32 trainable parameters, two FP32 Adam moment arrays require 256 bytes. The observed optimiser tensor storage is 264 bytes because the implementation also stores scalar step state for the two factor tensors. Parameters and gradients are additional allocations. This small discrepancy is a useful reminder that a parameter formula and a concrete object inventory have different scopes.

A future learning-rate schedule must be indexed by optimiser updates rather than microbatch calls. Chapter 10’s scheduling semantics remain the reference. Holding a constant rate here isolates the adapter/checkpoint mechanism; it does not demonstrate that scheduling is unnecessary for real adaptation.

    flowchart TD
    A["Clear gradients"] --> B["Backpropagate microbatch token sums S_k divided by N_update"]
    B --> C{"All gradients finite?"}
    C -- No --> F["Fail stop; keep current configuration"]
    C -- Yes --> D["Clip trainable gradient norm to 1"]
    D --> E["AdamW step; measure fixture loss"]
    E --> A
  

The loop above runs once per optimiser update with explicit failure exits. Notice that nonfinite gradients stop the run instead of silently changing precision, and that six such passes produce the toy’s small loss motion rather than any quality result.

Saving an adapter is saving a dependency

An adapter is not a standalone language model. Its meaning depends on the base weights, target modules, rank, scaling, tokenizer/template and training objective. A small file can be perfectly intact and entirely wrong for the model beside it.

The reference writes factor tensors in safetensors and a separate manifest binding the base digest, batch/mask identity, configuration, execution-source digest, environment and optimiser definition. Existing destinations are refused. The manifest distinguishes the random educational base from a pretrained model.

Loading checks the expected run identity and independently hashes the actual destination base. A copied manifest cannot make a different same-shaped base compatible. The loader also compares the destination model configuration and actual target layout, including rank, alpha and dropout. Same-shaped weights with a different attention configuration or adapter scale are refused. File bytes, factor keys, shapes, dtypes and adapter content are checked before copying tensors.

After reload, the reference evaluates both models on identical inputs. Their logits agree exactly in the recorded CPU run. Separate negative tests alter the file bytes, request a wrong base and omit the expected binding; each fails at its intended boundary.

The PEFT checkpoint-format documentation describes the corresponding separation of adapter weights and configuration in that library. Our educational manifest is not a claim of automatic PEFT or pretrained-checkpoint interoperability.

We save an adapter inference state, not a complete resumable training process. Exact resumption also needs optimiser/scheduler state, random generators, data position and relevant scaler state. Chapter 30 owns that recovery problem. Reloading logits alone must not be presented as evidence of exact training resume.

Keep numerical precision explicit

The executed CPU loop uses FP32. Its small size makes parameter freezing, gradients and state inventories easy to inspect.

The proposed pretrained path separates FP16 base/compute from FP32 adapters and optimiser state. The checkpoint is stored in BF16, so the first load performs a BF16-to-FP16 conversion; that is a declared numerical intervention, with FP16’s ±65,504 range against BF16’s approximately 3.4e38. It has not been executed. Actual module placement, autocast behaviour, loss scaling and supported kernels must be inspected before that description becomes evidence.

A dtype printed beside model configuration is not a complete numerical account. Loss reduction may use FP32 even when a matrix operation uses FP16. Some tensors may remain FP32 after other parameters are cast. A successful tiny FP16 matrix does not establish stable gradients for a complete language-model run.

Nonfinite gradients should produce an explicit failure with the current configuration retained. Automatically changing precision, rank or sequence length and reporting only the surviving run would hide a changed experiment.

Validation is not another training-loss column

A completed real study must validate on permitted development examples and retain its raw predictions. Decreasing training loss can coexist with invalid JSON, wrong policy routes, overfitting or loss of an earlier capability.

The reference loop deliberately makes no validation-quality or retention claim. The fixtures are available mechanism data. The incident retention role is also only a future probe set until an earlier capability and its comparison interface have been established.

Likewise, choosing the best checkpoint by repeatedly examining a protected test set turns that set into development information. Early stopping, prompt changes and adapter selection need a permitted validation boundary before execution.

The baseline protocol from Chapter 20 governs the eventual comparison. It requires the four engineering-set controls and preserves the distinction between target loss, canonical task metrics and trusted scorer evidence.

Run the mechanism and preserve its limits

The inspectable implementation lives in ftlab.adapters.sft. The permitted batch entry point is:

docker compose -f docker-compose.book.yml --profile training run --rm book-training-cpu python examples/part_iv/all_cpu.py --output /results/p4-cpu-NEW

Choose a new destination. It writes per-phase reports and ignored adapter artifacts, with source and configuration identities. The test suite checks single-shift loss semantics, token-weighted accumulation, parameter freezing, repeatable seeds, adapter identity and exact reload.

Pretrained acquisition, inference, training and evaluation remain NOT_RUN_AUTHORIZATION. The real-model service entry points refuse those operations before attempting network or model work. Passing the educational loop does not satisfy the canonical contract’s open rights, resource, scoring or release requirements.

We now have an executed chain from selected token masks through optimiser updates to a reloadable adapter bound to its base. The next chapter opens the low-rank layer itself: why its two matrices have these dimensions, how their gradients flow, and why an apparently harmless all-zero initialisation stops learning.

Further Reading

  1. Edward J. Hu et al., “LoRA: Low-Rank Adaptation of Large Language Models” (2021). Read the source. Freezes pretrained weights and learns rank-decomposed updates, defining the adapter whose identity the chapter binds.
  2. Hugging Face contributors, “SFT Trainer” (living documentation). Read the source. Documents the full SFT loop with PEFT and packing, clarifying what the verified minimal path deliberately excludes; also recommended in Chapter 16 for mask semantics.
  3. Hugging Face contributors, “PEFT checkpoint format” (v0.18.0 developer guide). Read the source. Separates adapter weights from base weights with an explicit config, motivating base-digest and config binding on reload.