← Fine Tuning From First Principles

Selecting a Starting Checkpoint

Pin the model, tokenizer and protocol; inspect four small Qwen candidates; and separate a reproducible investigation shortlist from evidence of task quality or training compatibility.

Two engineers select the same model family and obtain different results before either trains anything. One starts from a base checkpoint; the other starts from an instruction-tuned checkpoint. Their tokenizers also come from different repository revisions. A framework default adds a reasoning prefix to one prompt. The experiment log calls both systems “the small Qwen model”. That name cannot explain the difference, and it cannot reproduce either experiment.

Checkpoint selection begins by making the proposed system identifiable. Chapter 03 supplied the task contract; Chapter 04 supplied a conditional resource ledger. We now need the exact starting parameters, the mechanism that turns text into their inputs, and the evidence supporting their use. Choosing a plausible starting point is possible before a task benchmark exists. Establishing which starting point performs best requires that benchmark.

Our running task remains synthetic incident triage: read a fictional report, produce the contracted structured record, apply a versioned routing policy and abstain where the contract requires it. This task combines language understanding, output formatting and policy-dependent decisions. A checkpoint that already follows instructions may reduce formatting work. A base checkpoint may still be a useful control or a suitable starting point for a different adaptation objective. Neither proposition is a measured result here.

What the checkpoint actually identifies

An architecture describes a computation: embedding tables, attention projections, feed-forward blocks, normalisation and output mapping. Its configuration supplies dimensions and conventions. The weights are the learned numerical values used in that computation. Identical shapes permit very different functions because training changes those values.

A pretraining checkpoint is a saved state during or after pretraining. A released base model generally exposes the resulting text-continuation model without presenting it as the instruction-following endpoint. Continued pretraining applies further language-modelling updates, often to a particular language or domain. Instruction tuning changes the training distribution towards demonstrations of requested behaviour. A subsequent fine-tuned checkpoint may contain any of several adaptation histories. These labels describe provenance and intended use; a name does not disclose every dataset, optimisation decision or capability.

The artifact boundary also matters. An adapter checkpoint saves an additional learned parameter set and its configuration, usually relying on separately supplied base weights. A merged checkpoint incorporates a compatible adapter’s update into a particular base representation. A quantised inference artifact encodes parameters for a specified inference runtime, packing scheme and kernels. It does not establish that a training framework can update those tensors or that an adapter loader can reconstruct the intended computation. The PEFT checkpoint documentation describes why an adapter file alone is insufficient to reconstruct its base model.

Model configuration, tokenizer configuration and generation configuration have different jobs. Model configuration defines the network and positional assumptions. Tokenizer configuration helps define the mapping between text and token IDs, including special tokens and conversation rendering. Generation configuration supplies decoding defaults such as stopping IDs or sampling settings. Recording only the first leaves important experimental variables unspecified.

Checkpoint selection and training-method selection therefore remain separate decisions. The same pinned checkpoint could be considered for full updates or a frozen-base adapter. Either proposal needs its own state ledger, compatibility checks and evaluation. A smaller checkpoint can be easier to inspect and reproduce without being more accurate on our task.

Identity includes the input protocol

Let a proposed system be identified by

\[ I=(R_m,C_m,R_t,C_t,H_T,G,A), \]
where \(R_m,C_m\) identify the model repository and immutable commit, \(R_t,C_t\) identify the tokenizer repository and commit, \(H_T\) identifies the chat-template bytes, \(G\) records generation and rendering choices, and \(A\) records adaptation assumptions. A branch such as `main` can resolve to different commits tomorrow. Resolving it once and recording the returned commit prevents that particular substitution; it does not authenticate the publisher or prove the artifact is suitable.

A content hash answers a different question: have these recorded bytes changed? FineTuneLab reuses the project JSON encoding and SHA-256 from the task-contract component. Keys are sorted, insignificant JSON whitespace is removed and non-finite values are rejected. This is the project’s deterministic encoding, not a claim to implement every JSON canonicalisation standard. Resealing a changed document creates a different identity. It does not validate the changed claims.

The manifest validator checks internal structure and consistency. A separate source-binding check compares the recorded configuration, template, weight inventory and licence identity with captured upstream files. This separation is useful: a syntactically valid manifest can still describe the wrong repository. Neither check is a digital signature. The captured source provenance remains part of the trust boundary.

For this first inspector, model and tokenizer must share the exact repository and commit. Cross-repository tokenizers are rejected until an explicit compatibility procedure exists. Sharing a family name or a vocabulary size would not establish that token ID 42 has the same meaning, or that an embedding row was learned for the same symbol.

Checkpoint identity: what must be pinned

The figure expands the identity tuple into its seven pinned components and shows the four candidates as frozen revisions rather than family names. The verdict is deliberately an abstention: precise identity earns a place on the shortlist, not a claim about task quality.

Four small candidates, frozen before comparison

On 17 November 2026 we inspected the authoritative repositories for Qwen3-0.6B, Qwen3-0.6B-Base, Qwen2.5-0.5B-Instruct and Qwen2.5-0.5B. The following commits, rather than their mutable repository tips, define this inspection.

Candidate Immutable revision Classification
Qwen3-0.6B c1899de289a04d12100db370d81485cdf75e47ca Instruct
Qwen3-0.6B-Base da87bfb608c14b7cf20ba1ce41287e8de496c0cd Base
Qwen2.5-0.5B-Instruct 7ae557604adf67be50417f59c2c2f167def9a775 Instruct
Qwen2.5-0.5B 060db6499f32faf8b98477b0a26969ef7d8b9987 Base

The collection script allowed these four repositories and an explicit list of metadata, configuration, licence and tokenizer files. It imposed a 40 MiB total response limit, a 256 KiB small-file limit and a 12 MiB tokenizer-JSON limit, with a separate 1 MiB API-response bound. It received 32,693,599 bytes. No weight URL was requested. Tokenizer JSON stayed in the ignored local cache; smaller captured files, their URLs, byte counts and hashes are retained in research/ch05/upstream/.

All four captured listings identify Apache-2.0, are ungated, and list safetensors weights. These are repository observations. We did not read weight headers, load tensors or establish actual loaded dtypes. The model names are not parameter measurements.

Metadata or derivation Qwen3 pair Qwen2.5 pair
Configured architecture Qwen3ForCausalLM Qwen2ForCausalLM
Layers / hidden width / MLP width 28 / 1,024 / 3,072 24 / 896 / 4,864
Attention heads / KV heads / head dimension 16 / 8 / 128 14 / 2 / 64
Configured embedding rows 151,936 151,936
Input/output embedding tying Declared true Declared true
Logical unique parameters, derived 596,049,920 494,032,768
Tokenizer entries including added tokens 151,669 151,665

Both inspected families use rotary position configuration, RMS normalisation and gated SiLU feed-forward blocks. Their attention dimensions differ materially. In Qwen3 the query projection has 2,048 output coordinates despite a hidden width of 1,024. Assuming every projection is square would undercount its adapter. FineTuneLab derives dimensions from the declared heads and head dimension, supported by inspection of the installed Transformers implementation; actual module creation is still untested.

A disagreement worth retaining

For the reviewed dense implementations, write hidden width as \(H\), intermediate width as \(J\), layer count as \(L\), vocabulary rows as \(V\), query width as \(Q=n_h d\), and key/value width as \(K=n_{kv}d\). With tied embeddings the logical unique count is

\[ P=VH+L\left(2HQ+2HK+3HJ+2H+N\right)+H. \]
The first two attention terms count query, key, value and output projection matrices; \(3HJ\) counts the three feed-forward matrices; \(2H\) counts the two layer-normalisation scales; and the final \(H\) counts the final normalisation. In these definitions, \(N=2d\) for Qwen3's query/key normalisation scales and \(N=Q+2K\) for Qwen2's query/key/value biases. Untied output embeddings would add \(VH\). This ledger must reject unsupported architecture variants rather than apply the formula by resemblance.

The Qwen2.5 Hub tensor count agrees with the derived 494,032,768. Qwen3-0.6B-Base agrees with 596,049,920. Qwen3-0.6B instead reports 751,632,384 stored tensor parameters in the captured API metadata, although both Qwen3 configurations declare tied embeddings. The difference is exactly \(151,936\times1,024=155,582,464\), one embedding-shaped matrix. That arithmetic suggests a question about storage and tying; it does not establish the answer. Weight inspection and loader behaviour are needed to resolve it.

The listed safetensors files are respectively 1,503,300,328 bytes for Qwen3-0.6B, 1,192,135,096 for its Base counterpart, and 988,097,824 for each Qwen2.5 candidate. File bytes include representation and serialization details. They cannot settle which tensors become shared objects after loading.

The resource screen therefore uses the larger of the configuration-derived and Hub counts, explicitly labelled as a conservative metadata representation count. It is not a measurement of actual loaded parameters, and it is not a bound on all training memory. Keeping the discrepancy visible is more useful than silently replacing one number with the other.

The tokenizer is part of the model interface

A causal language model continues token sequences. Conversation roles, message boundaries and an assistant-generation prefix must be represented in those sequences. A template translates a structured conversation into the textual control format expected by a checkpoint; tokenisation then maps that representation to IDs. The Transformers chat-template guide explains this interface and the role of generation prompts.

The captured tokenizers report 151,643 base vocabulary entries. Added control tokens raise the actual tokenizer lengths to the values in the table. Model embedding tables have 151,936 rows, so those three numbers describe different boundaries rather than an automatic error. Valid special IDs must still fit the tokenizer and embedding table. Replacing a tokenizer merely because the total sizes match would be an invalid shortcut.

The base tokenizers use <|endoftext|> as EOS, ID 151643; the instruction candidates use <|im_end|>, ID 151645. Padding uses <|endoftext|> in all four. The tokenizer BOS property is unset even though the model configurations name BOS ID 151643. That difference must be interpreted through actual encoding behaviour, rather than “corrected” by adding a token everywhere. Later training must also distinguish real EOS targets from padding in its loss mask. A shared padding and EOS ID makes that distinction especially important.

We executed only the cached tokenizers, using Transformers 5.17.0 and tokenizers 0.23.1, local_files_only=True and trust_remote_code=False. No model was instantiated. The fictional user message was:

Fictional credentials request. Return a structured record.

With an assistant-generation prompt, the recorded default renders produce 19 tokens for each Qwen3 candidate, 30 for Qwen2.5 Base and 40 for Qwen2.5 Instruct. These lengths describe this exact message and template, not general token efficiency. The Qwen2.5 defaults insert different system text. Rendering to text and then encoding with add_special_tokens=False reproduces each recorded template-tokenisation sequence. Exact texts, IDs, arguments and timings are in research/ch05/template-inspection-002.json.

All four repositories contain chat templates, including the base releases. Template availability is therefore insufficient evidence of instruction training. A base tokenizer can render a conversation without its weights having learned to follow that conversation protocol reliably.

Pin thinking mode before observing outputs

For both inspected Qwen3 tokenizers, the default and explicit enable_thinking=True renders coincide on this input. Explicit enable_thinking=False changes the continuation prefix: an empty <think>…</think> block is appended, increasing the prefix to 23 tokens. This is observed template behaviour, not generated reasoning or evidence about answer quality. The pinned Qwen3 model card documents the mode control.

Our proposed Qwen3 chat baseline fixes enable_thinking=False. Allowing a framework default to choose the mode would change the experiment while leaving the checkpoint name unchanged. If a later study investigates thinking mode, it needs a separate protocol identity, output budget and latency accounting. It must not compare a training intervention and a reasoning-mode intervention as though only weights changed.

Position limits are not a training budget

Every captured tokenizer declares model_max_length=131072. The Qwen3-0.6B model configuration instead declares 40,960 positions; its Base counterpart and both Qwen2.5 configurations declare 32,768. We preserve the disagreement rather than select the largest advertised number. No long-context execution was performed.

The analytical proposal uses sequence length 512 and microbatch one. This shorter declared working length leaves activation memory unknown; it does not prove a run will fit. A later data protocol must count the complete rendered input and output, including delimiters and prefixes. An over-length case should be rejected or processed by a declared truncation policy. Silent truncation could remove the evidence needed for the label and make the checkpoint comparison meaningless.

Licence and framework evidence have boundaries

The captured licence texts and model-card identifiers agree on Apache-2.0. The selector accepts that identified licence under its declared allowlist. This does not settle the rights to the incident data, a derived dataset or an intended distribution. The draft task contract’s model-rights gate remains unverified.

Distribution obligations include providing the licence, retaining applicable notices, marking modified files and handling relevant NOTICE material; patent and trademark provisions also need consideration for the actual use. The authoritative licence text governs those conditions. FineTuneLab records them and their source identity; it does not issue a legal determination about a future product.

Likewise, a model card’s framework recommendation is a compatibility claim with a scope. The captured Qwen3 card recommends Transformers 4.51.0 or later; the Qwen2.5 card warns about releases before 4.37.0. A configuration’s transformers_version is creation metadata, not a complete supported-environment specification. Our successful tokenizer execution establishes only that these tokenizers ran on the recorded local versions.

The historical environment reports PyTorch 2.6.0+cu124, an RTX 2060 with compute capability 7.5 and no native BF16 support. Several planned adaptation libraries were absent. Architecture recognition by Transformers cannot establish native-Windows quantisation-kernel support, PEFT interoperability or a successful backward pass. LoRA and QLoRA compatibility remain unverified.

Connect the identity to the resource ledger

For separate rank-eight adapters on query and value projections, the additional parameter count is

\[ P_A=Lr\left[(H+Q)+(H+K)\right]. \]
This gives 1,146,880 parameters for each Qwen3 configuration and 540,672 for each Qwen2.5 configuration. The arithmetic assumes the declared target projections and excludes bias, embedding and output-head training. These targets are source-supported proposals. Actual module names, tensor shapes, adapter insertion and trainability must later be checked on an instantiated model.

The default screen declares FP16 base/adapter storage and compute, FP32 gradients and two FP32 optimizer-state tensors, no separate master copy, and unknown activation, workspace and allocator requirements. This is an explicit proposed representation change from the repositories’ BF16 storage metadata; no weight conversion has occurred. A BF16 requirement is rejected against the captured native capability, with no automatic fallback to FP16.

For Qwen3-0.6B the selected base representation contributes 1,503,264,768 analytical bytes. Adapter weights contribute 2,293,760, gradients 4,587,520 and optimizer states 9,175,040. Their subtotal is 1,519,321,088 bytes. The other Qwen3 candidate has a smaller selected base count; each Qwen2.5 proposal uses its own dimensions and count. None of these subtotals includes unknown device terms. In particular, they are not measured peak allocations.

The historical GPU inventory reports total capacity 12,288 MiB and free memory 8,118 MiB at its capture time. The selector deliberately discards the old free-memory value for a future proposal and retains capacity only as the Chapter 04 fallback. It also refuses to inherit illustrative “supported” runtime declarations as tested compatibility. The candidate reports consequently return INSUFFICIENT_EVIDENCE, identify current free memory as unknown and require a fresh check immediately before any future authorised canary.

If an engineer proposes training biases or the output head in place while supplying only this separate-adapter ledger, the selector returns UNSUPPORTED_IN_PLACE_LEDGER. Those base tensors need their own gradients, optimizer states and possible precision copies. Quietly turning the proposal into ordinary LoRA would answer a different question. The diagnostic preserves the requested method and explains the missing accounting.

Implement a shortlist that can abstain

The reusable ftlab.checkpoints component loads a versioned manifest, validates its identity, derives the supported architecture ledger, checks adapter/base identity and invokes ftlab.budget.assess. Its packaged schema fixes runtime claims to UNKNOWN, NOT_RUN or NOT_MEASURED. This metadata inspector has no path that can turn those fields into successful training evidence.

Run the offline demonstration from the repository root:

python examples/ch05/checkpoint_demo.py --output experiments/FT-05/new-run.json

Use a fresh output path to preserve earlier records. It checks all four captured candidates and exercises deliberately incompatible probes: a synthetic disallowed licence, a BF16 requirement, missing template evidence, an incompatible adapter architecture and an unsupported in-place ledger. Those modified probes are fictional negatives; they are not claims about the four upstream licences or templates.

The returned shortlist contains all four candidates for further investigation and no preferred winner. Its decision is ABSTAIN_PENDING_BASELINES. Hard exclusions and missing evidence have separate fields. There is no weighted “quality score”: architecture names, download popularity and parameter counts cannot supply accuracy evidence. The established task-contract readiness report is included unchanged; training and launch remain false regardless of shortlist membership.

Adapter identity checking requires the exact base repository/revision, tokenizer identity, template hash and architecture. Even a successful identity match leaves loading unverified. Later code must additionally inspect target shapes, adapter configuration, saved modules and actual trainable tensors. Two base models with equal counts do not become interchangeable hosts for a learned adapter.

The automated tests cover mutable and missing revisions, invalid repository IDs, mismatched tokenizers, invalid EOS/padding IDs, unsupported architectures, non-integral count units, altered templates, inconsistent ledgers, source disagreement, licence exclusions and unknown runtime evidence. They also verify that a manifest cannot claim executed training and that historical free memory is discarded. Earlier decision, contract and budget tests remain regression controls. Frozen template-record checks protect the recorded experiment relationships; they are distinct from the live tokenizer execution that produced those records.

Prepare a comparison capable of changing the choice

Chapter 06 receives a preparation manifest, not an evaluated winner. It proposes all four exact candidates, preserves the task-contract hash and fixes decoding choices before any generated answer is seen. Base candidates receive a declared raw completion protocol; instruction candidates receive their native chat templates. Qwen3 chat explicitly disables thinking. The full rendered prompts and token budgets must be frozen with the later data protocol.

This is a comparison of usable starting systems, including their protocols. It cannot isolate the causal effect of instruction tuning because protocol and training history differ together. A later matched same-weights protocol control could investigate that distinction, but it must be separately identified rather than added after inspecting holdout failures.

The future baseline must measure schema validity, task correctness, policy consistency, required abstention, retention, latency and memory under the same synthetic case definitions and scoring rules. Invalid JSON, truncations and missing responses remain visible. A deterministic policy control and the strongest admissible no-training alternatives belong in the comparison. Validation may guide protocol development; the evaluation holdout cannot become a source of examples or prompt repairs.

The preparation file leaves dataset hashes, statistical adequacy, acceptance freeze, dependency validation and execution authorisation unresolved. Chapter 18 still owns the benchmark freeze in the book’s dependency plan. Nothing in this chapter authorises weight acquisition or a GPU experiment.

The result we have earned is precise: four reproducible metadata identities, executed tokenizer inspections, model-specific analytical screens and a selector that explains its abstention. Which candidate best serves incident triage, and whether any proposed adaptation runs on this computer, remain questions for separately authorised evidence.

Further Reading

  1. Hugging Face contributors, “PEFT checkpoint format” (living documentation). Read the source. Documents why an adapter file alone cannot reconstruct a pinned system, since base-model identity lives outside the adapter artefact.
  2. Hugging Face contributors, “Chat templates” (living documentation). Read the source. Explains the template mechanics behind the thinking-mode finding, where identical text yields different training boundaries.
  3. Apache Software Foundation, “Apache License, Version 2.0” (2004). Read the source. The authoritative text behind the licence gate, listing redistribution obligations while saying nothing about dataset rights.