← Fine Tuning From First Principles

Long Context, Multimodal Models and Mixture of Experts

Treat long context, multimodal projection and mixture-of-experts routing as three separate mechanisms, derive what each changes in the forward pass, check them on tiny CPU fixtures, and show why a compatibility check that refuses unsupported combinations is the honest substitute for runs that were never made.

The tool trajectories in “Tool Use, Code, Structured Outputs and Agent Data” were bounded at 8 events. Real systems keep asking for more: longer documents in the prompt, images beside the text, and models so large that only part of them runs for each token. Three families of model answer those demands. Long- context models extend the sequence a transformer can attend over. Multimodal models feed non-text inputs into a language model. Mixture-of-experts models route each token to a subset of specialised sub-networks.

These are often discussed together because they arrive together in model releases. They are not one technique. Each changes a different part of the forward pass, and each breaks a different assumption that an adapter, a training script or an evaluation relied on. This chapter treats them separately. For each, it derives what changes, runs the smallest honest version on a CPU, and states what cannot be concluded from it. It closes with the question that ties them together: which adaptations remain valid when context length, input modality or expert routing changes?

The experiments here are deliberately small, and the chapter says so at every step. No long-context, vision-language or mixture-of-experts model was loaded, downloaded or trained, and no GPU was used. That gap is recorded as NOT_RUN_AUTHORIZATION in proposal P8-45C, and nothing below should be read as a result about such models.

Mechanism one: attention over a longer sequence

In the transformer, each attention head computes a score between every query position and every key position. For a batch of one sequence of length L and H heads, the score tensor has

$$ N_{\text{scores}} = H \cdot L^2 $$
elements, and the work to produce and apply them grows in proportion to L² times the head width. Doubling L quadruples the score count. This is the reason long-context work concentrates on attention: Dao and colleagues open the [FlashAttention](https://arxiv.org/abs/2205.14135) paper by noting that self-attention's time and memory are quadratic in sequence length, and Beltagy and colleagues make the same point as the motivation for [Longformer](https://arxiv.org/abs/2004.05150).

Two masks act on the scores before the softmax. The causal mask removes every key position j greater than the query position i, so a token cannot attend to the future. The padding mask removes keys at padded positions, so filler tokens added to make a batch rectangular contribute nothing. After masking, every valid row of attention weights sums to one.

Fine-tuning adds a third constraint that is easy to violate silently. Labels mark ignored positions with −100, as in Chapter 16, “Which Tokens Receive Loss?”. If an example is longer than the context limit and is truncated, the supervised positions near the end can be cut off. The tensor still has a valid shape. The batch still runs. The example has zero targets: under a loss averaged over all targets in the batch it contributes nothing while still counting as a training example, and under a loss averaged per example it produces 0/0, which is not a number. For a long-context task, where the answer typically comes after the long input, this is the expected failure, not an edge case.

The executed probe

Record FT-P8-INTEGRATED-02-P8-45A-s37 runs real forward passes through a randomly initialised tiny transformer with width 8, 2 heads and a hidden size of 16, on a CPU with one thread. At L = 16 and L = 32 the attention score tensors have 512 and 2048 elements: 2 × 16² and 2 × 32². The output shape is [1, L, 262] for the 262-token byte vocabulary. In both cases the last two positions are padding, and one position is supervised.

The record confirms, from the stored attention weights, that every weight above the diagonal is exactly zero, that every weight on a padded key is exactly zero, and that replacing the padded tokens with different values leaves the logits at valid positions unchanged. The authoring cross-check recomputed these facts from the full attention arrays in the record: the largest future weight and the largest padding-key weight are both zero, and every valid row sums to one within 10⁻⁵. One caution about the logit check: because the padding sits at the last two positions, the causal mask alone already stops earlier positions from seeing it, so unchanged logits confirm causality rather than the padding mask. The padding mask is shown directly by the zero weight from the last padded query to the padded key before it.

The record also contains two timings: 0.001814 seconds for L = 16 and 0.000952 seconds for L = 32. The longer sequence ran faster. That is not a contradiction of the quadratic count. Each timing covers two forward passes, the traced one and the one with altered padding, and the first measurement probably includes warm-up; the record does not separate the causes. Timings of forward passes this small are dominated by overheads that have nothing to do with attention. No scaling law can be fitted from these two numbers, and none is claimed. The quadratic count is a mathematical property of the tensor, and the record verifies the count, not the latency.

Three requests in the probe were refused rather than run. A context of 64 is refused by the existing model configuration with MODEL_BUDGET; that limit is a safety boundary inherited from earlier chapters, and it was kept. A compatibility declaration asking for context 64 is refused with COMPATIBILITY_BUDGET. And a synthetic example whose 16 labels contain one target at position 15 is refused when truncated to 8, with TARGET_TRUNCATED, because the only target would be lost. None of these is a successful 64-token run.

Mechanism two: a projection into the language model

A language model consumes a sequence of vectors. Text becomes vectors through a token embedding. An image can become vectors by cutting it into patches, flattening each patch and multiplying by a learned matrix. That is the approach the Vision Transformer applied to sequences of image patches. Vision-language systems such as LLaVA then connect a vision encoder to a language model and train the combination end to end.

The projection step, stripped of everything learned, is a matrix product. For a flattened patch x and a projection W with as many rows as the target width,

$$ y = W x . $$
Record **FT-P8-INTEGRATED-02-P8-45B-s37** executes exactly this. A 2 × 2 scalar patch, flattened to x = (1, 2, 3, 4), is multiplied by a supplied 2 × 4 matrix whose rows are (1, 0, 0, 1) and (0, 1, 1, 0). The first output is 1 + 4 = 5 and the second is 2 + 3 = 5, so y = (5, 5). The implementation refuses inputs of the wrong shape with `PATCH_SHAPE` and non-finite values with `PATCH_FINITE`.

The point of so small an example is what it lacks. The matrix was written down, not learned. There is no image encoder, no pretraining, no alignment between the projected vectors and the language model’s embedding space, and no evaluation of any visual task. A shape-valid projection is not a learned vision-language model. It shows only where the interface sits. For adaptation, that interface is where most compatibility questions arise: which component an adapter targets (the vision encoder, the projection or the language model), whether the image processor’s normalisation matches what the encoder was trained with, and whether the position scheme accounts for the image tokens. None of those questions can be answered by a projection fixture, and the compatibility check refuses a real vision-language family with UNSUPPORTED_BACKEND.

Mechanism three: routing tokens to experts

A sparsely gated mixture-of-experts layer replaces a single feed-forward block with several experts and a router that sends each input to a small subset of them. The model can hold many parameters while each token uses only a few. Switch Transformers simplified the router to send each token to a single expert. Because routing is uneven, each expert has a capacity: the tokens in a batch divided evenly across experts, multiplied by a capacity factor. In the Switch design, a token that arrives at a full expert is not processed by it and passes to the next layer through the residual connection.

The routing rule for one token with expert scores r₁, …, r_E and top-1 selection is

$$ k^\ast = \arg\max_k r_k, \qquad \text{admit if } \text{load}_{k^\ast} < C, \text{ otherwise drop}. $$
P8-45B executes this rule with two experts and capacity C = 2. Three tokens have scores (2, 1), (3, 0) and (4, 1). All three prefer expert 0. The first two are admitted, filling it. The third finds expert 0 full and is dropped; it is not sent to expert 1. The loads are (2, 0) and one token is dropped. Ties go to the lower index. Non-finite scores fail with `ROUTER_SCORES`, and a request for more than one expert per token fails with `ROUTER_CAPACITY`, because only top-1 is implemented.

This example says nothing about load balance, which in a trained model emerges from training and auxiliary losses. Here every token was constructed to prefer the same expert. What the example does show is that capacity is an execution fact that changes which computation a token receives. For adaptation, that has a direct consequence: an adapter attached to one expert only affects the tokens routed to it, and changing the routing, whether by fine-tuning the router or by changing the batch composition, changes which tokens the adapter sees. A fine-tuning report for a mixture-of-experts model that omits routing and drop statistics cannot be interpreted.

Compatibility as the honest substitute

The three mechanisms share one engineering lesson. Each has parameters that, if they do not match the assumptions of the training code, produce something that runs and is wrong. A context limit below the supervised positions gives a batch with no targets. A projection with the wrong normalisation gives vectors the language model has never seen. An adapter on the wrong expert trains on almost nothing. None of these raises an exception by default.

FineTuneLab’s answer is an exact, versioned capability declaration, checked before anything runs. A request must state eight fields: family, modality, context, adapter target, rank, precision, device and router. The admission function compares them with a fixed matrix:

Family Modality Contexts Adapter target Ranks Precision Device Router
tiny-transformer tokens 16, 32 blocks.0.project 1, 2 float32 cpu dense
docket structured 1 linear 1 float64 cpu dense
patch-reference numeric-patch 4 none 0 float64 cpu none
router-reference numeric 3 none 0 float64 cpu top1-capacity-drop

Any family not in the matrix is refused with UNSUPPORTED_BACKEND. A mismatched modality, device or router is refused with its own code. An unsupported context, rank or precision gives COMPATIBILITY_BUDGET, and an unknown adapter target gives COMPATIBILITY_TARGET. Declarations with missing or extra fields give COMPATIBILITY_CONFIG. The check never guesses from a model name.

Three mechanisms and the admission matrix

The figure summarises what each mechanism changes (quadratic score count, linear projection, capacity-gated routing) and the exact matrix that admits only declared CPU fixtures. It marks real long-context, vision-language and mixture-of-experts families as refused, with their probes not run.

Admission is a narrow statement. An admitted declaration is within this implementation’s CPU bounds. It does not authorise a dataset, a training run or a deployment, and it does not imply that a real model with the same nominal properties would behave as the fixture does.

What a real-model bridge would have to check

Admitting a real long-context, multimodal or mixture-of-experts model would need more fields than the matrix has, and it is worth stating them, because each one corresponds to a silent failure.

For long context, the declaration would need the position scheme the model was trained with and the longest length at which it was evaluated, the truncation side, and a count of supervised targets after truncation for every training example. A model that accepts a longer input than it was trained on still produces output; whether that output is any good is an evaluation question, not a shape question.

For multimodal input, it would need the identity of the image processor and its normalisation, the encoder and projection versions, the number of tokens each image becomes, and which of encoder, projection and language model each adapter targets. A mismatch in any of these produces vectors of the right shape.

For experts, it would need the routing rule, the capacity factor, any auxiliary balancing loss, whether the router itself is trained, and which experts carry adapters. It would also need routing statistics recorded during training and evaluation, because the same adapter on the same expert sees different tokens under a different routing.

Across all three, multi-device execution adds its own assumptions about how sequences, experts or layers are split between devices, which Chapter 29, “Scaling Beyond One GPU”, introduced. None of this is implemented here, and the honest state of FineTuneLab is that such models are refused.

Negative controls

The implementation refuses each mismatched field of a capability declaration with its own code. The focused tests mutate seven of the eight fields (not the router) and assert only that an error is raised, not which code. They also pass patches of the wrong shape and with non-finite values, non-finite router scores and label sequences whose only target is removed by truncation. The refusal of top-2 routing is in the code but has no dedicated test. The authoring cross-check recomputed the projection and the routing assignments independently, counted the attention elements from the stored weight arrays, and verified the masks and the truncation target counts from the recorded labels.

What the experiments show and do not show

They show that the tiny transformer’s attention score count is H · L², that its causal and padding masks hold exactly in the recorded forwards, and that the truncation guard refuses an example whose only target would be lost. They show the arithmetic of a linear patch projection and of top-1 routing with capacity and drops. They show that the compatibility check refuses every combination outside a small, explicit matrix.

They do not show GPU behaviour, latency scaling or memory scaling. The process high-water mark in the record, 234912 KiB, is the resident set of the whole process, not the memory used by attention. They do not show the quality of any long-context model, any vision-language model or any trained mixture of experts. No learned router was trained, and no real model of any of the three kinds was run.

Interpretation

Each mechanism moves the boundary of what a fine-tuning recipe must check. Long context moves the risk into truncation and cost: the targets at the end of a long example are the first to be cut, and the cost grows with the square of the length. Multimodal input moves the risk into interfaces: the projection and the processor must match what the components were trained with. Expert routing moves the risk into execution accounting: which tokens reach which parameters is itself a variable. A training script that was correct for a dense text model can be silently wrong for all three.

Tooling for these families changes quickly, and any list of supported combinations goes stale. The durable practice is the one shown here: declare the combination exactly, check it against what has actually been verified, and refuse everything else with a specific reason.

Limitations

The tiny transformer is randomly initialised. The contexts are 16 and 32. The timings are single runs that include warm-up. The projection is supplied and the router is constructed. The compatibility matrix lists only the fixtures in this package. The proposed real-model probes are NOT_RUN_AUTHORIZATION.

Practical consequences

  • Count supervised targets after truncation, and refuse examples that lose them.
  • Budget attention by H · L², and measure latency and memory on the real hardware rather than inferring them from the count.
  • For multimodal adaptation, record which component each adapter targets, and check processor and normalisation settings against the encoder’s training configuration.
  • For mixture-of-experts adaptation, report routing, load and drop statistics alongside task results.
  • Admit model, context, target, rank, precision, device and router by exact declaration, never by model name, and refuse anything unverified.

FineTuneLab

ftlab.capstone.compatibility provides MATRIX, admit, patch, route, truncate_labels and the experiment used by P8-45A. They reuse the tiny transformer and causal loss from earlier parts. None of them loads a real model, and the installed ftlab command does not expose them as profiles.

Where this leads

Chapters 43 to 45 extended FineTuneLab in three directions and found the same discipline in each: declare what is being done, check it against what is supported, measure at the right boundary and refuse the rest. The remaining chapters assemble that discipline into something a reader can install and run. The next chapter, “FineTuneLab: Design the Usable Solution”, describes the command-line interface, its configuration, its contracts and the typed graph of stages behind every run.

References

Further Reading

  1. Jianlin Su et al., “RoFormer: Enhanced Transformer with Rotary Position Embedding” (2021). Read the source. Rotary position embeddings with the length and decay properties a real long-context declaration must record.
  2. Dmitry Lepikhin et al., “GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding” (2020). Read the source. Production mixture-of-experts showing routing, load and capacity as execution facts an adapter report must include.
  3. Jean-Baptiste Alayrac et al., “Flamingo: a Visual Language Model for Few-Shot Learning” (2022). Read the source. A real vision-language interface showing why a shape-valid projection is not a learned multimodal model.