# SICQL — Building a Model From Q, V and Policy Networks
# Book: Models From First Principles
# Chapter URL: https://programmer.ie/books/models-from-first-principles/04-chapter/
# Prose fingerprint (sha256, normalized): 050677a4afb3c2d3646244c27494774f29205f15c10412d01c96c470b7603c47
# Snapshot generated at build time by Hugo. This is the complete rendered prose
# of the chapter; it is not a transcript of the source file.

SICQL — Building a Model From Q, V and Policy Networks In the previous post we took the MR.Q idea and expanded it into something richer.
Instead of asking one question of a shared representation, EBT asked several:
How good is this state-action pair? -> Q How good is the state more generally? -> V What action should be preferred? -> Policy How much better is Q than V? -> Advantage That already gave us a more expressive system.
But there is another architectural step we can take.
We can stop treating those outputs as anonymous branches inside one large model and make them first-class models in their own right.
That is the central idea of this post.
We are going to build a SICQL-style architecture where the system is visibly composed from smaller models:
context embedding + candidate embedding ↓ encoder ↓ zsa ┌─────────┼─────────┐ ↓ ↓ ↓ QHead VHead PolicyHead ↓ ↓ ↓ Q V action logits └──── Q - V ────────┘ advantage The important lesson is not the acronym.
The important lesson is composition.
A model can be built from models.
And once those components are explicit, we can inspect them, test them, train them, replace them and reason about them independently.
1. Why make the heads explicit? At first glance, these two designs may look almost identical.
Inline design class Model(nn.Module): def __init__(self, d): super().__init__() self.encoder = nn.Linear(d * 2, 256) self.q = nn.Linear(256, 1) self.v = nn.Linear(256, 1) self.pi = nn.Linear(256, 3) Component design class Model(nn.Module): def __init__(self, encoder, q_head, v_head, policy_head): super().__init__() self.encoder = encoder self.q_head = q_head self.v_head = v_head self.policy_head = policy_head Mathematically, either might represent the same functions.
Architecturally, they are not the same.
The second design gives names and boundaries to the components.
That matters because a boundary gives us something we can test.
input contract ↓ component ↓ output contract If QHead promises:
[B, Z] -> [B] we can test that promise without involving the rest of the system.
If PolicyHead promises:
[B, Z] -> [B, A] we can replace it without rewriting the encoder.
The architecture becomes easier to reason about because the code now reflects the conceptual model.
2. Start with the latent representation We will assume that we already have two embeddings:
context_emb # [B, D] candidate_emb # [B, D] A simple pair encoder can concatenate them and project them into a latent state-action representation.
import torch from torch import nn class PairEncoder(nn.Module): def __init__(self, embedding_dim: int, latent_dim: int): super().__init__() self.net = nn.Sequential( nn.Linear(embedding_dim * 2, latent_dim), nn.ReLU(), nn.LayerNorm(latent_dim), nn.Linear(latent_dim, latent_dim), nn.ReLU(), ) def forward(self, context, candidate): assert context.shape == candidate.shape assert context.dim() == 2 pair = torch.cat([context, candidate], dim=-1) return self.net(pair) Shape flow:
context [B, D] candidate [B, D] ↓ concatenate pair [B, 2D] ↓ encoder zsa [B, Z] We will call that final representation zsa.
The name is useful shorthand for encoded state-action information, but remember the rule from the previous posts:
A variable name does not prove semantics.
If both context and candidate are present, then zsa is certainly a pair representation.
Whether it deserves a stronger reinforcement-learning interpretation depends on how the data and objective are defined.
3. Build QHead as its own model The Q head predicts one scalar from the latent representation.
class QHead(nn.Module): def __init__(self, latent_dim: int, hidden_dim: int): super().__init__() self.net = nn.Sequential( nn.Linear(latent_dim, hidden_dim), nn.ReLU(), nn.Linear(hidden_dim, 1), ) def forward(self, zsa): return self.net(zsa).squeeze(-1) That is the entire model.
zsa [B, Z] ↓ Linear(Z -> H) ↓ ReLU ↓ Linear(H -> 1) ↓ Q [B] Nothing mystical happened.
We simply gave the scalar predictor a name and a contract.
Now we can test it independently.
def test_q_head_shape(): head = QHead(latent_dim=64, hidden_dim=32) z = torch.randn(8, 64) q = head(z) assert q.shape == (8,) That small test has real value.
It tells us that if the composed model later emits the wrong Q shape, the error is probably outside the Q head.
4. Build VHead independently The V head can have exactly the same physical architecture while representing a different learned function.
class VHead(nn.Module): def __init__(self, latent_dim: int, hidden_dim: int): super().__init__() self.net = nn.Sequential( nn.Linear(latent_dim, hidden_dim), nn.ReLU(), nn.Linear(hidden_dim, 1), ) def forward(self, z): return self.net(z).squeeze(-1) Notice something important.
The code does not know whether this is Q or V.
Physically both heads are:
vector ↓ MLP ↓ scalar Their meaning comes from:
which inputs they receive + which target they are trained against + which loss is applied This is one of the most important recurring ideas in model architecture.
Two networks may be structurally identical while learning entirely different functions.
5. The state-only V question returns In the EBT post we raised a subtle issue.
Suppose the encoder receives:
context + candidate and then V receives the resulting pair representation.
Then the path is:
(context, candidate) ↓ zsa ↓ VHead ↓ value That value is candidate-conditioned.
Calling it a pure V(s) may therefore be too strong.
A stricter architecture uses two representations:
context ───────────────> state encoder ──> zs ──> VHead context + candidate ───> pair encoder ───> zsa ─> QHead └─────> PolicyHead We can implement that explicitly.
class StateEncoder(nn.Module): def __init__(self, embedding_dim, latent_dim): super().__init__() self.net = nn.Sequential( nn.Linear(embedding_dim, latent_dim), nn.ReLU(), nn.LayerNorm(latent_dim), ) def forward(self, context): return self.net(context) Then:
zs = state_encoder(context) zsa = pair_encoder(context, candidate) v = v_head(zs) q = q_head(zsa) Now the architectural contract matches the conventional notation more closely:
V(s) Q(s, a) Does that automatically make the model better?
No.
It makes the semantics cleaner.
Whether it improves the actual task is an empirical question.
That distinction matters.
6. Build PolicyHead as a classifier The policy head predicts logits over actions.
class PolicyHead(nn.Module): def __init__(self, latent_dim: int, hidden_dim: int, num_actions: int): super().__init__() self.net = nn.Sequential( nn.Linear(latent_dim, hidden_dim), nn.ReLU(), nn.Linear(hidden_dim, num_actions), ) def forward(self, zsa): return self.net(zsa) Shape flow:
zsa [B, Z] ↓ PolicyHead ↓ action logits [B, A] If there are three possible actions:
reject revise accept then:
logits.shape == [B, 3] We deliberately return logits, not probabilities.
For training with cross entropy:
loss_policy = F.cross_entropy(action_logits, action_targets) PyTorch expects raw logits.
If we need probabilities for inspection:
probs = action_logits.softmax(dim=-1) Again, separate the training interface from the presentation interface.
7. Compose the model Now we have four independent pieces:
PairEncoder QHead VHead PolicyHead We can compose them.
class InContextQModel(nn.Module): def __init__(self, encoder, q_head, v_head, policy_head): super().__init__() self.encoder = encoder self.q_head = q_head self.v_head = v_head self.policy_head = policy_head def forward(self, context, candidate): zsa = self.encoder(context, candidate) q = self.q_head(zsa) v = self.v_head(zsa) policy_logits = self.policy_head(zsa) advantage = q - v return { "zsa": zsa, "q": q, "v": v, "advantage": advantage, "policy_logits": policy_logits, } The whole architecture is now visually obvious.
PairEncoder │ zsa ┌───────────┼───────────┐ │ │ │ ▼ ▼ ▼ QHead VHead PolicyHead │ │ │ q v logits │ │ └──────┬────┘ ▼ advantage This is what building a model from models looks like in ordinary PyTorch.
8. Why component boundaries are valuable Once the heads are separate modules, several engineering practices become much easier.
Independent initialization q_head = QHead(128, 64) v_head = VHead(128, 64) policy_head = PolicyHead(128, 64, 3) Independent tests def test_policy_head(): head = PolicyHead(128, 64, 3) z = torch.randn(4, 128) logits = head(z) assert logits.shape == (4, 3) Independent checkpointing torch.save(model.q_head.state_dict(), "q_head.pt") torch.save(model.v_head.state_dict(), "v_head.pt") Independent replacement model.q_head = BiggerQHead(...) Independent freezing for p in model.v_head.parameters(): p.requires_grad = False Independent optimization optimizer = torch.optim.AdamW([ {"params": model.encoder.parameters(), "lr": 1e-4}, {"params": model.q_head.parameters(), "lr": 3e-4}, {"params": model.v_head.parameters(), "lr": 3e-4}, {"params": model.policy_head.parameters(), "lr": 1e-4}, ]) Those are not merely code-organization conveniences.
They create experimental control.
9. First-class heads enable clean ablations Suppose we want to know whether the policy objective actually helps Q ranking.
With explicit heads we can train:
Experiment A: encoder + Q Experiment B: encoder + Q + V Experiment C: encoder + Q + V + Policy Then compare the metric we actually care about.
results = { "q_only": evaluate(model_q), "q_v": evaluate(model_qv), "q_v_policy": evaluate(model_full), } This is much stronger than assuming:
more heads = more intelligence.
Perhaps the policy head regularizes the representation and improves ranking.
Perhaps it causes gradient conflict and makes Q worse.
Perhaps it does nothing.
The architecture gives us a hypothesis.
The experiment gives us the answer.
10. Explicit components reveal parameter ownership A surprisingly common debugging problem is not knowing which parameters belong to which conceptual subsystem.
With first-class heads:
for name, p in model.named_parameters(): print(name, tuple(p.shape)) might produce:
encoder.net.0.weight encoder.net.0.bias encoder.net.2.weight ... q_head.net.0.weight q_head.net.0.bias ... v_head.net.0.weight ... policy_head.net.0.weight ... The namespace itself becomes documentation.
We can count parameters by subsystem.
def count_params(module): return sum(p.numel() for p in module.parameters()) print("encoder:", count_params(model.encoder)) print("Q:", count_params(model.q_head)) print("V:", count_params(model.v_head)) print("policy:", count_params(model.policy_head)) This lets us answer a useful question:
Where did the model complexity actually go?
Often the shared encoder dominates.
The extra heads may be cheap.
11. Initialize the heads deliberately Because the heads are independent modules, we can make initialization explicit too.
def init_linear(module): if isinstance(module, nn.Linear): nn.init.xavier_normal_(module.weight) nn.init.zeros_(module.bias) q_head.apply(init_linear) v_head.apply(init_linear) policy_head.apply(init_linear) Now initialization is not an accidental side effect of construction.
It is part of the model contract.
That matters especially when loading partial checkpoints or comparing experiments.
12. A full standalone SICQL-style model Here is the whole system in one place.
import torch from torch import nn import torch.nn.functional as F class PairEncoder(nn.Module): def __init__(self, embedding_dim, latent_dim): super().__init__() self.net = nn.Sequential( nn.Linear(embedding_dim * 2, latent_dim), nn.ReLU(), nn.LayerNorm(latent_dim), nn.Linear(latent_dim, latent_dim), nn.ReLU(), ) def forward(self, context, candidate): assert context.dim() == 2 assert context.shape == candidate.shape return self.net(torch.cat([context, candidate], dim=-1)) class ScalarHead(nn.Module): def __init__(self, latent_dim, hidden_dim): super().__init__() self.net = nn.Sequential( nn.Linear(latent_dim, hidden_dim), nn.ReLU(), nn.Linear(hidden_dim, 1), ) def forward(self, z): return self.net(z).squeeze(-1) class PolicyHead(nn.Module): def __init__(self, latent_dim, hidden_dim, num_actions): super().__init__() self.net = nn.Sequential( nn.Linear(latent_dim, hidden_dim), nn.ReLU(), nn.Linear(hidden_dim, num_actions), ) def forward(self, z): return self.net(z) class SICQLStyleModel(nn.Module): def __init__(self, embedding_dim=128, latent_dim=128, hidden_dim=64, num_actions=3): super().__init__() self.encoder = PairEncoder(embedding_dim, latent_dim) self.q_head = ScalarHead(latent_dim, hidden_dim) self.v_head = ScalarHead(latent_dim, hidden_dim) self.policy_head = PolicyHead(latent_dim, hidden_dim, num_actions) def forward(self, context, candidate): zsa = self.encoder(context, candidate) q = self.q_head(zsa) v = self.v_head(zsa) policy_logits = self.policy_head(zsa) return { "q": q, "v": v, "advantage": q - v, "policy_logits": policy_logits, "zsa": zsa, } Run it:
model = SICQLStyleModel() context = torch.randn(16, 128) candidate = torch.randn(16, 128) out = model(context, candidate) for key, value in out.items(): print(key, value.shape) Expected:
q [16] v [16] advantage [16] policy_logits [16, 3] zsa [16, 128] At this point we have an inspectable composed model.
Now we need to decide how to train it.
13. Q objective: pairwise preference Assume we have:
context preferred candidate rejected candidate We want:
Q(context, preferred) > Q(context, rejected) A stable pairwise logistic objective is:
def pairwise_q_loss(q_good, q_bad): return F.softplus(-(q_good - q_bad)).mean() Training:
good = model(context, preferred) bad = model(context, rejected) loss_q = pairwise_q_loss(good["q"], bad["q"]) The quantity being optimized is the difference.
That means absolute Q values are not automatically calibrated.
Again:
ranking and calibration are different problems.
14. V objective: expectile regression A common way to train a value estimate is expectile regression.
Define the residual:
residual = target_q - v Then give asymmetric weight to positive and negative residuals.
def expectile_loss(v, target_q, tau=0.7): diff = target_q.detach() - v weight = torch.where(diff > 0, tau, 1 - tau) return (weight * diff.square()).mean() The parameter tau controls which part of the Q distribution the V model is pushed toward.
For example:
tau = 0.5 -> ordinary squared-error center tau > 0.5 -> places more weight on underestimation The exact semantics depend on the training setup.
The important point for this series is architectural:
V has its own objective.
The fact that Q and V share an encoder does not make them the same model.
15. Policy objective Suppose action labels are integers:
0 = reject 1 = revise 2 = accept Then:
loss_policy = F.cross_entropy( out["policy_logits"], action_targets, ) If we have behavioral or preference weights, the policy objective could be more sophisticated.
But the basic architecture is simply a classifier over the shared representation.
16. Compose the losses A simple multi-task objective could be:
loss = ( loss_q + 0.5 * loss_v + 0.2 * loss_policy ) These coefficients are not magic constants.
They are hyperparameters expressing how strongly each task influences the shared representation.
That immediately creates another experimental question:
Are the tasks helping each other or fighting each other?
17. Measure gradient conflict between heads Because the encoder is shared, each objective sends gradients into the same parameters.
We can inspect the direction of those gradients.
def flat_grad(loss, params): grads = torch.autograd.grad( loss, params, retain_graph=True, allow_unused=True, ) parts = [] for p, g in zip(params, grads): if g is None: parts.append(torch.zeros_like(p).flatten()) else: parts.append(g.flatten()) return torch.cat(parts) Then:
params = list(model.encoder.parameters()) g_q = flat_grad(loss_q, params) g_v = flat_grad(loss_v, params) g_pi = flat_grad(loss_policy, params) Cosine similarity:
def cosine(a, b): return F.cosine_similarity(a, b, dim=0).item() print("Q vs V:", cosine(g_q, g_v)) print("Q vs policy:", cosine(g_q, g_pi)) print("V vs policy:", cosine(g_v, g_pi)) Interpretation:
positive -> gradients broadly cooperate near zero -> mostly unrelated negative -> objectives push encoder in opposing directions This does not automatically tell us what to do.
It tells us what is happening.
That is the evidence-first approach.
18. Advantage is a derived quantity We compute:
advantage = q - v That operation has no parameters.
Q model ──┐ ├── subtraction -> advantage V model ──┘ This distinction matters.
Advantage is not another neural network.
It is a derived signal produced by two learned models.
If advantage behaves badly, the cause is upstream.
That gives us a useful debugging path:
bad advantage ↓ inspect Q ↓ inspect V ↓ inspect shared representation ↓ inspect objectives/data 19. Detach advantage or not? Sometimes advantage is used only for telemetry or downstream decision logic.
Then we may want:
advantage = (q - v).detach() That makes a strong statement:
Nothing downstream of advantage is allowed to send gradients back through Q or V.
Compare:
advantage = q - v versus:
advantage = (q - v).detach() These have identical numeric values in the forward pass.
But their optimization behavior is different.
This is exactly the sort of small line that an LLM can generate correctly-looking code around while changing the training semantics dramatically.
When in doubt:
print(advantage.requires_grad) print(advantage.grad_fn) Runtime evidence wins.
20. Policy weights are not policy probabilities An easy interpretability mistake is to look directly at the final linear layer weights and treat them as the policy.
Suppose:
last = model.policy_head.net[-1] print(last.weight.shape) The weight matrix might be:
[A, H] Those are model parameters.
They are not the action probabilities for a specific example.
The policy for an example is:
logits = model(...)["policy_logits"] probs = logits.softmax(dim=-1) Parameters tell us how the model transforms representations.
Outputs tell us what it predicts for the current input.
Do not confuse the two.
21. Checkpoint the components separately Explicit composition makes partial checkpoints natural.
torch.save({ "encoder": model.encoder.state_dict(), "q_head": model.q_head.state_dict(), "v_head": model.v_head.state_dict(), "policy_head": model.policy_head.state_dict(), }, "sicql_components.pt") Loading:
state = torch.load("sicql_components.pt", map_location="cpu") model.encoder.load_state_dict(state["encoder"]) model.q_head.load_state_dict(state["q_head"]) model.v_head.load_state_dict(state["v_head"]) model.policy_head.load_state_dict(state["policy_head"]) This lets us ask more interesting questions.
For example:
What happens if we keep the encoder fixed but retrain Q? What happens if we transfer Q and replace the policy head? What happens if V is trained on a different dataset? Those experiments are easy because the architecture has boundaries.
22. But component checkpoints create compatibility contracts Modularity creates flexibility.
It also creates compatibility problems.
Suppose an old Q head expects:
latent_dim = 128 but the new encoder outputs:
latent_dim = 256 Then:
q = old_q_head(new_encoder_output) fails.
Or worse, suppose the dimensions match but the meaning of the representation changed because the encoder was retrained.
The shapes still work.
The semantics may not.
A robust checkpoint should therefore store metadata.
checkpoint = { "model_version": 3, "embedding_model": "example-embed-v2", "embedding_dim": 768, "latent_dim": 128, "encoder_version": "pair-v4", "head_versions": { "q": "q-v2", "v": "v-v1", "policy": "pi-v3", }, "state_dict": model.state_dict(), } Compatibility is more than tensor shape.
23. Test components before testing the full model A useful test ladder is:
1. individual layer 2. individual head 3. encoder 4. composed forward pass 5. individual loss 6. combined loss 7. backward pass 8. optimizer update 9. tiny-batch overfit 10. validation behavior For example:
def test_model_contract(): model = SICQLStyleModel( embedding_dim=32, latent_dim=16, hidden_dim=8, num_actions=3, ) context = torch.randn(5, 32) candidate = torch.randn(5, 32) out = model(context, candidate) assert out["q"].shape == (5,) assert out["v"].shape == (5,) assert out["advantage"].shape == (5,) assert out["policy_logits"].shape == (5, 3) assert out["zsa"].shape == (5, 16) This turns architecture documentation into executable code.
24. Verify every subsystem receives gradients After one backward pass:
loss.backward() we can inspect each component.
def grad_summary(module): total = 0.0 missing = [] for name, p in module.named_parameters(): if p.grad is None: missing.append(name) else: total += p.grad.norm().item() return total, missing Then:
for name, module in { "encoder": model.encoder, "q": model.q_head, "v": model.v_head, "policy": model.policy_head, }.items(): norm, missing = grad_summary(module) print(name, norm, missing) This gives us an immediate answer to:
Did every component actually participate in training?
A head can exist in the architecture and still be functionally disconnected from the loss.
25. Verify parameters actually move Gradient presence is not enough.
Save a copy before step().
before = { name: p.detach().clone() for name, p in model.named_parameters() } optimizer.step() Then:
for name, p in model.named_parameters(): delta = (p.detach() - before[name]).abs().max().item() print(name, delta) Now we know which components changed.
This catches mistakes such as:
parameters omitted from the optimizer; frozen modules; zero gradients; stale optimizer references after replacing a head. That last one is especially important.
26. Replacing a head can invalidate the optimizer Suppose we build the optimizer:
optimizer = torch.optim.AdamW(model.parameters(), lr=1e-3) Then later replace Q:
model.q_head = BiggerQHead(...) The new Q head’s parameters are not automatically added to the existing optimizer.
The optimizer still holds references to the old parameter objects.
Check ownership:
opt_ids = { id(p) for group in optimizer.param_groups for p in group["params"] } for name, p in model.q_head.named_parameters(): print(name, id(p) in opt_ids) If false, rebuild the optimizer or update its parameter groups.
This is a perfect example of why architectural modularity and runtime debugging belong together.
27. Swap Q without changing the rest of the model Now we can exploit the interface.
class DeeperQHead(nn.Module): def __init__(self, latent_dim, hidden_dim): super().__init__() self.net = nn.Sequential( nn.Linear(latent_dim, hidden_dim), nn.GELU(), nn.Linear(hidden_dim, hidden_dim), nn.GELU(), nn.Linear(hidden_dim, 1), ) def forward(self, z): return self.net(z).squeeze(-1) Replacement:
model.q_head = DeeperQHead(128, 64) Nothing else needs to change because the interface remains:
[B, 128] -> [B] That is the power of the component boundary.
28. Replace the encoder instead The same principle works higher in the hierarchy.
Our model only requires the encoder to satisfy:
(context, candidate) -> zsa So we could replace concatenation with richer interaction features.
class InteractionEncoder(nn.Module): def __init__(self, d, z): super().__init__() self.net = nn.Sequential( nn.Linear(d * 4, z), nn.ReLU(), nn.LayerNorm(z), ) def forward(self, context, candidate): features = torch.cat([ context, candidate, context - candidate, context * candidate, ], dim=-1) return self.net(features) Then:
model.encoder = InteractionEncoder(128, 128) The heads do not care.
This is architecture as substitution.
29. Stable interfaces let us run controlled experiments We can compare:
Encoder A: concat Encoder B: concat + difference + product Encoder C: attention-based while keeping:
QHead VHead PolicyHead training objective optimizer training data fixed.
That is a clean encoder ablation.
Similarly we can compare head architectures while keeping the encoder fixed.
Modularity makes scientific control easier.
30. Candidate reranking A common use case is one context with many candidate responses.
Suppose:
context.shape == [B, D] candidates.shape == [B, K, D] We can repeat the context:
B, K, D = candidates.shape ctx = context[:, None, :].expand(B, K, D) ctx = ctx.reshape(B * K, D) cand = candidates.reshape(B * K, D) Run all candidates:
out = model(ctx, cand) q = out["q"].reshape(B, K) Choose the highest Q:
best_idx = q.argmax(dim=-1) But now SICQL gives us more than Q.
We can also inspect:
v = out["v"].reshape(B, K) adv = out["advantage"].reshape(B, K) policy = out["policy_logits"].reshape(B, K, -1) The system has become a diagnostic surface, not merely a ranker.
31. Multiple outputs can disagree Suppose candidate A has:
high Q small advantage policy says revise and candidate B has:
slightly lower Q large advantage policy says accept Which candidate wins?
The neural architecture cannot answer that question for us.
That is a decision policy layered on top of the model outputs.
For example:
def decision_score(out): accept_prob = out["policy_logits"].softmax(-1)[..., 2] return ( 0.7 * out["q"] + 0.2 * out["advantage"] + 0.1 * accept_prob ) Those weights are another hypothesis.
They require validation.
The model architecture produces signals.
The application decides how to use them.
Keep those layers conceptually separate.
32. Architecture vs objective vs decision logic At this point we can distinguish three levels clearly.
Architecture encoder QHead VHead PolicyHead Training objectives pairwise Q loss expectile V loss policy cross entropy Runtime decision logic rank by Q rank by advantage accept if policy probability > threshold combine signals These are different things.
A lot of model confusion comes from mixing them together.
33. A small synthetic training experiment Let’s build a toy task so the complete system can be executed.
We will generate contexts and candidates where the preferred candidate is the one more aligned with the context.
def make_batch(batch_size=256, dim=32): context = torch.randn(batch_size, dim) good = context + 0.25 * torch.randn(batch_size, dim) bad = -context + 0.25 * torch.randn(batch_size, dim) action_good = torch.full((batch_size,), 2, dtype=torch.long) action_bad = torch.zeros(batch_size, dtype=torch.long) return context, good, bad, action_good, action_bad Build the model:
model = SICQLStyleModel( embedding_dim=32, latent_dim=64, hidden_dim=32, num_actions=3, ) optimizer = torch.optim.AdamW(model.parameters(), lr=1e-3) Training loop:
for step in range(1000): context, good, bad, a_good, a_bad = make_batch() out_good = model(context, good) out_bad = model(context, bad) loss_q = pairwise_q_loss(out_good["q"], out_bad["q"]) target_q = torch.cat([ out_good["q"].detach(), out_bad["q"].detach(), ]) pred_v = torch.cat([ out_good["v"], out_bad["v"], ]) loss_v = expectile_loss(pred_v, target_q, tau=0.7) policy_logits = torch.cat([ out_good["policy_logits"], out_bad["policy_logits"], ]) action_targets = torch.cat([a_good, a_bad]) loss_policy = F.cross_entropy(policy_logits, action_targets) loss = loss_q + 0.5 * loss_v + 0.2 * loss_policy optimizer.zero_grad(set_to_none=True) loss.backward() optimizer.step() if step % 100 == 0: with torch.no_grad(): rank_acc = (out_good["q"] > out_bad["q"]).float().mean() print( step, float(loss), float(rank_acc), ) This is not intended as a benchmark.
It is an executable architecture demonstration.
34. Run the baseline first Before celebrating the composed model, measure a trivial baseline.
Because our synthetic good candidates are deliberately close to the context, cosine similarity may solve the task almost perfectly.
def cosine_score(context, candidate): return F.cosine_similarity(context, candidate, dim=-1) Evaluate:
base_good = cosine_score(context, good) base_bad = cosine_score(context, bad) baseline_acc = (base_good > base_bad).float().mean() If the simple baseline wins, that is not embarrassing.
It tells us the neural model is unnecessary for that dataset.
Complexity should earn its place.
35. A component should justify itself For every component ask:
What new capability is this intended to provide? What metric should improve if it works? What baseline does it need to beat? What ablation isolates its contribution? For VHead:
Does it improve downstream policy learning? Does its advantage signal correlate with useful decisions? Does removing it change performance? For PolicyHead:
Does policy supervision improve the shared representation? Does it hurt Q ranking? Does the policy itself predict useful actions? For the pair encoder:
Does learned interaction outperform cosine similarity or a linear probe? That is how architecture becomes evidence rather than decoration.
36. The deeper lesson: software architecture matters inside neural architecture SICQL introduces something beyond another mathematical idea.
It introduces software architecture into model architecture.
The conceptual system says:
Q V Policy The code says:
QHead VHead PolicyHead That alignment matters.
When conceptual boundaries and code boundaries match, we gain:
clearer tests; clearer checkpoints; clearer metrics; cleaner ablations; easier replacement; easier debugging; easier reuse. A model is not only a function.
It is also a piece of software.
37. When not to split everything into modules There is a danger in taking modularity too far.
You could turn every linear layer into its own class.
LinearOne ActivationOne LinearTwo OutputWrapper That would not necessarily improve comprehension.
A useful component boundary should correspond to something meaningful:
separate objective separate semantic role separate interface separate experiment separate checkpoint separate replacement candidate QHead qualifies.
A single ReLU usually does not.
Good modularity compresses thought.
Bad modularity creates ceremony.
38. Why SICQL still isn’t enough We have now built a model composed from several models.
But all of them still operate in one forward pass.
The pattern is:
input ↓ encoder ↓ heads ↓ outputs There is no persistent internal state that is refined over several reasoning steps.
The network does not repeatedly revisit a latent representation.
That creates the next architectural question:
What if a single feed-forward transformation is not enough?
We can introduce recurrence.
But we can go further than one recurrent state.
We can create two coupled states operating at different scales:
low-level state ↕ high-level state The low-level state can update several times while the high-level state changes more slowly.
That is the architectural move we will make next.
What we learned SICQL gave us another layer of decomposition.
We started with:
one model Then discovered:
encoder Q model V model policy model And each of those decomposed again into:
Linear activation Linear The important advance was not simply adding more layers.
It was making the conceptual components explicit.
That gave us:
independent tests independent losses independent checkpoints independent replacement clean ablations parameter ownership clear debugging paths The model became more complex mathematically while becoming easier to inspect structurally.
That is a useful architecture.
In the next post we make a much larger jump.
Next: HRM — Hierarchical Reasoning With Fast and Slow Recurrent State Instead of evaluating the representation once, we will build two coupled recurrent systems:
input ↓ low-level recurrent processing ↓ repeated local refinement ↓ high-level recurrent update ↓ repeat That is where Models From First Principles moves from composed feed-forward models into iterative reasoning architectures.