Continued Pretraining Versus Instruction Tuning
Separate learning a domain's token distribution from learning an instruction-to-response mapping and from answering with facts that change, then read a controlled experiment's negative results as carefully as its positive ones.
“We want the model to know our domain.” That sentence hides at least three different requirements. The model might need to assign high probability to the kind of text the domain produces: its vocabulary, its formats, its recurring phrases. It might need to map a particular kind of request to a particular kind of response. Or it might need to answer with facts that are true today, such as which team currently owns a service.
These requirements call for different training objectives, and in the third case possibly for no training at all. Continued pretraining on domain text optimises the first. Instruction tuning on prompt-response pairs optimises the second. Neither optimises the third, because a fact that changes after training is not in the training data.
This chapter makes the distinction precise, then runs a small controlled experiment in which the objectives are varied while the model, the update count and the optimiser are held fixed. Several of its results are negative or mixed. They are reported as such, because they are the most instructive part.
The results below have also been executed in the pinned Docker CPU environment as FT-P5-INTEGRATED-02-27, using the repaired Part IV foundation and local FineTuneLab 0.12.0. The integrated evidence map records source, configuration, environment and artifact hashes for each claim. Earlier staging and authoring comparisons remain historical provenance; they are not independent acceptance. No pretrained-model operation is authorised by these runs.
Three requirements, three objectives
Let a training sequence be x_0, …, x_(T-1). Chapter 08 derived next-token loss with a single causal shift: the logit at position t-1 predicts the label at position t. Chapter 16 introduced the target mask m_t, which decides which positions contribute to the loss without removing them from the conditioning context. With J the set of selected target positions,
- Continued pretraining selects every position of domain documents. It estimates the domain’s token distribution.
- Response-only instruction tuning selects only response positions of prompt-response pairs. It estimates the conditional distribution of a response given its prompt.
- Prompt-only training selects only prompt positions of the same pairs. It is a deliberate mismatch control: it trains on instruction data but asks the model to predict the wrong part.
The sequence log-probability decomposes exactly as
The figure shows which positions each objective supervises and why a fact created after training must arrive via the input. Notice that the mask decides what is learned, and that only x can carry revision p2.
Staleness is an information problem
The third requirement needs a different argument. Suppose a routing policy maps an entity to a team, and the policy changes from revision p1 to revision p2 after training. A model’s answer is a function of its parameters and its input, y = f_θ(x). If neither θ nor x contains p2, the answer can equal p2’s route only by coincidence.
Training cannot place p2 in θ if p2 did not exist when θ was trained. Only the input can carry it. That is what retrieval-augmented generation, introduced for knowledge-intensive tasks by Lewis and colleagues, and tool lookup both do: they put the current fact into x at inference time. Lewis and colleagues explicitly name updating a model’s world knowledge as an open problem that non-parametric memory can help address.
This gives a sharp rule. A model can be trained to use current information supplied in its input. It cannot be trained to possess information created after training. When the requirement is the second, the remedy is a lookup, and the right comparison is between a model given the current fact and the lookup itself, not between a stale model and a fresh one.
Design the controlled experiment
The experiment uses a new artificial grammar of twelve tokens, unrelated to the incident-triage data of earlier chapters. Two kinds of record exist.
Domain records alternate two tokens: amber with copper, or teal with zinc. Training sequences contain two repetitions, such as
bos amber copper amber copper eos
Evaluation sequences contain three repetitions, so the domain probe asks whether the alternation continues one step further than training ever showed.
Response records contain a query, an entity, a policy marker, a route and an answer:
bos query amber policy north answer north eos
The response is the route repeated after answer, followed by eos. The task is
to copy the route the prompt supplies. Evaluation adds a prefix token at the
start. This is a held-out surface variant, not an independent semantic family;
with two training records and two evaluation records, the experiment can expose
mechanisms but cannot estimate a population rate.
Every method starts from the same random weights for a given seed: a one-block causal Transformer with width 16, two heads and MLP width 32. Every training condition takes 80 full-batch AdamW updates at learning rate 0.015 with gradient clipping at norm 1, in FP32 on one CPU thread. The conditions are:
| Condition | Training records | Objective | Targets per update |
|---|---|---|---|
| None | none | not trained | 0 |
| Continued pretraining | domain | all next tokens | 10 |
| Response tuning | response | response only | 4 |
| Prompt-only mismatch | response | prompt only | 10 |
Equal update counts are not equal training signal. Response tuning sees four target tokens per update; the other two conditions see ten. The table states this rather than hiding it.
The mean in the loss also changes how hard each token pulls. Because L divides by |J|, the gradient is
Seeds 37 and 38 were run separately. All measurements are recorded as raw logits per target position, so every aggregate can be recomputed from the evidence by a scalar log-sum-exp.
Training loss has an irreducible floor
Before reading the held-out results, look at where training loss stopped. After 80 updates, the mean training NLL just before the last update was 0.1395 for continued pretraining at seed 37, 0.1396 for prompt-only training and 0.0005 for response tuning.
The first two numbers are not under-training. In each domain training sequence,
the token after bos is amber or teal with equal frequency, and nothing before
it distinguishes them. The best possible prediction assigns probability 1/2 to
each, contributing ln 2 ≈ 0.693 to that position’s NLL. Two such positions among
ten targets give a floor of
Read the held-out results
The held-out measurements are domain NLL (mean over the 14 target positions of
the two longer domain sequences), response NLL (teacher-forced, mean over the 4
response targets of the prefixed evaluation records) and exact response accuracy
(greedy generation of two tokens, which must equal the gold route and eos).
| Seed | Condition | Domain NLL | Response NLL | Exact responses |
|---|---|---|---|---|
| 37 | None | 2.622700 | 2.774758 | 0/2 |
| 37 | Continued pretraining | 2.298619 | 5.379908 | 0/2 |
| 37 | Response tuning | 5.202149 | 5.355251 | 0/2 |
| 37 | Prompt-only mismatch | 7.642333 | 7.341755 | 0/2 |
| 38 | None | 2.488214 | 2.255055 | 0/2 |
| 38 | Continued pretraining | 2.126450 | 5.346413 | 0/2 |
| 38 | Response tuning | 8.936643 | 1.058768 | 1/2 |
| 38 | Prompt-only mismatch | 8.568787 | 6.045480 | 0/2 |
Four observations follow, each with its limit.
Continued pretraining lowered domain NLL at both seeds, by about 0.32 and 0.36. It also raised response NLL and produced no exact responses. Learning the domain’s distribution did not teach the copying task.
Response tuning fitted its training pairs almost perfectly but failed on the
held-out variant. At seed 37 it generated eos first and the route second, for
both records: the right tokens in the wrong order. At seed 38 it answered one
record exactly and gave the wrong route for the other. A training NLL of 0.0005
did not transfer to one extra leading token.
Prompt-only training damaged both measurements. It learned the prompt distribution and left the response conditional unconstrained, as the decomposition predicts. This is the cleanest demonstration in the experiment that the loss mask, not the data, decides what is learned.
Response tuning’s domain NLL rose sharply, to 5.20 and 8.94. This is not forgetting: the response-tuned model started from random weights and was never trained on the domain. The domain probe here measures an untouched capability, not a lost one. Retention needs a different design, which follows below.
A mean can improve while the probe fails
The domain-NLL improvement deserves a closer look at the raw per-position values.
After continued pretraining at seed 37, the NLL at positions 2 to 4 of each
evaluation sequence was about 0.001: the model had learned the alternation
within the training length. Position 1 was about 0.69, the ln 2 floor. But at
position 5, where the training sequences had eos and the evaluation sequences
continue the pattern, the NLL was 8.47 for amber and 8.34 for teal. The model had
learned that sequences end after two repetitions, and it was confidently wrong.
The mean still improved, because twelve good positions outweighed two catastrophic ones. A reader who saw only “domain perplexity fell from 13.77 to 9.96” would conclude that continued pretraining helped the domain probe. The probe was designed to ask whether the pattern extends, and on exactly that question the model failed.
The independent length probes also failed on shorter sequences at their end token. The observed behaviour includes memorisation of the training sequence length in both directions, rather than reliable domain-pattern extrapolation.
This is a general hazard of averaged token metrics. They aggregate positions with different meanings. The remedy is not to distrust averages but to define the probe’s critical positions in advance and report them alongside the mean.
Measure retention from a learned state
To ask whether instruction tuning loses a domain capability, the capability must exist first. The retention branch therefore starts from the continued-pretraining state and applies 80 further updates under two conditions:
- Response adaptation: response-only tuning on the response records.
- Domain continuation control: 80 more updates of continued pretraining on the same domain records.
The control matters. Without it, any change in the domain probe would be attributed to the instruction data, when part of it might come from simply training longer.
| Seed | Learned domain NLL | After response adaptation | After domain continuation | Exact responses after adaptation |
|---|---|---|---|---|
| 37 | 2.298619 | 2.410589 | 2.544311 | 1/2 |
| 38 | 2.126450 | 4.748379 | 2.519719 | 2/2 |
At seed 38, response adaptation degraded the domain probe far more than the continuation control did: from 2.13 to 4.75, against 2.52. That pattern is consistent with interference between the objectives.
At seed 37, both branches degraded the probe, and the control degraded it more. Continued training on two short sequences reinforced the length pattern that the longer evaluation sequences violate. At this seed the experiment does not isolate any interference effect from response adaptation, and the result is recorded as inconclusive rather than averaged away.
One more observation is suggestive and nothing more. Starting response adaptation from the continued-pretraining state produced 1/2 and 2/2 exact responses, against 0/2 and 1/2 from random weights. With two seeds and two evaluation records, this cannot support a claim that continued pretraining helps later instruction tuning.
Separate stale inputs from stale models
The policy probe gives each trained model inputs in the training format and asks for the route under the current revision p2. Two kinds of input are supplied: one carrying the stale p1 route and one carrying the current p2 route.
For the response-tuned model at seed 37, the result was what the information argument predicts. Given the stale route, it copied the stale route for both entities: 0/2 correct under the current policy. Given the current route, it copied that and answered 2/2. At seed 38 it answered 1/2 with current input and 0/2 with stale input. The untrained model’s predictions coincided with the current route for one entity at seed 37; such coincidences are why chance baselines are always reported.
A lookup of the current policy table answered 2/2 at both seeds without any training. That is not a victory of lookup over learning. The lookup was given the current fact; the stale-input model was not. The experiment’s conclusion is the one the argument already established: when the requirement is the current fact, the input must carry it. Training a model to use supplied context is a legitimate objective; expecting it to know a revision it never saw is not.
What the experiment establishes
Within its small world, the experiment establishes that:
- the three objectives select different targets from the same records and produce visibly different models from the same starting weights;
- lower domain NLL can coexist with zero task accuracy and with confident failure at the probe’s critical positions;
- near-zero training loss on two records did not transfer to a one-token surface variant at seed 37;
- the mismatched mask degraded both measurements;
- retention effects need a learned starting capability and a continuation control, and with both in place the two seeds disagreed.
It does not establish that continued pretraining helps or harms instruction tuning in general, that any of these methods would behave similarly on a pretrained model, or that the observed effect sizes have any meaning beyond this grammar. Gururangan and colleagues studied domain-adaptive and task-adaptive pretraining on pretrained language models; this experiment is not a reproduction of their setting, and their findings are not attributed to it.
Limitations
The model is random and tiny. There are two seeds, two training records per kind and two evaluation records per probe. The response evaluation uses one surface variant and two-token greedy generation. Equal update counts give unequal target-token budgets. No pretrained model and no incident data are used.
The experiment was recorded in the engineering staging environment and then re-executed in a separate authoring container against the repaired Part IV package. Every discrete outcome (exact counts, route counts, generated tokens, policy-probe answers and training identities) matched exactly. The aggregate NLL values of the main conditions agreed to within 1.5 × 10⁻⁵, which leaves every value in the table unchanged at four decimal places. The retention branches, which run 160 updates from initialisation, drifted further: the seed-37 continuation control read 2.544311 in staging and 2.543629 in the authoring run, a difference of 7 × 10⁻⁴. The direction of every comparison in the retention table was unchanged. The final-state digests differed, as expected when two library builds round differently over many updates. The integrated run reproduces the staging table values. The larger authoring drift remains a historical cross-environment observation, not a universal numerical tolerance.
FineTuneLab
The experiment is implemented in the module ftlab.advanced.objectives,
integrated in the local 0.12.0 package. corpus declares every record, role and evaluation
question before any update; batch builds labels for an explicit objective using
the existing ftlab.learning.loss.causal_terms shift, so no second shift exists;
token_measurement records per-position logits; response_predictions performs
greedy generation; policy_probes builds the stale and current inputs; and
validate_run recomputes every aggregate from raw logits and refuses evidence
whose seed, corpus, source, starting state or training identities do not match.
The module is included in the locally verified 0.12.0 wheel; no release
or independent acceptance is implied.
Where this leads
Every condition in this chapter used one learning rate, one update count and no selection rule. The retention branch showed that how long training continues can matter as much as what it trains on. The next chapter, Hyperparameters, Overfitting and Forgetting, makes those choices the subject: schedules, effective batch, regularisation, validation-based selection, and a cleaner retention experiment in which forgetting is possible but not forced.
Further Reading
- Suchin Gururangan et al., “Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks” (2020). Read the source. Continues pretraining on domain and task data with gains across four domains. Read it as the pretrained-scale counterpart to the chapter’s continued-pretraining branch.
- Patrick Lewis et al., “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks” (2020). Read the source. Combines parametric memory with a Wikipedia index and names knowledge updating an open problem. Read it for the non-parametric remedy to the staleness the chapter argues training alone cannot supply; also recommended in Chapter 2 for the trained-retrieval distinction.
- Jason Wei et al., “Finetuned Language Models Are Zero-Shot Learners” (2021). Read the source. Instruction-tunes on dozens of datasets and surpasses zero-shot GPT-3 on most evaluations. Read it as the canonical response-mapping objective contrasted with continued pretraining; also recommended in Chapter 15 for format conversion.