Example

Count the parameters a mixture layer actually uses

Route a four-token batch through four experts, count the selected experts, and compare active parameters with the layer's total.

A mixture-of-experts layer contains many expert networks but evaluates only the top-k selected by a router for each token. The model’s total parameter count includes every expert; the active count for a token includes the router plus the selected experts. Those are different numbers, and confusing them turns a sparse model into an apparently dense one.

This example builds a layer with four experts of dimension 8 and routes a batch of four tokens with top-k 2. The router has 8 * 4 = 32 weights. Each expert has 8 * 8 = 64 weights. Active parameters per token are 32 + 2 * 64 = 160; total parameters are 32 + 4 * 64 = 288. The run also prints which experts each token selected, so the arithmetic can be checked against the routing decisions.

Run it

From the canonical repository at commit f958967d70992d7b09d9c75000bd8c543f1644d4:

python examples/ch19/moe.py --dim 8 --experts 4 --top-k 2 --json

Python 3.11 or later with PyTorch. CPU-only, seeded, no model download, network, GPU or credentials. It finishes in about a second.

Implementation

The router produces one logit per expert. Softmax turns them into weights, top-k selects the experts, and the selected weights are renormalised before their outputs are summed. The code iterates over the batch, so each token’s selection is explicit.

class MoELayer(nn.Module):
    def __init__(self, dim: int, num_experts: int, top_k: int = 2) -> None:
        super().__init__()
        if top_k > num_experts:
            raise ValueError("top_k must not exceed num_experts")
        self.dim = dim
        self.num_experts = num_experts
        self.top_k = top_k
        self.router = nn.Linear(dim, num_experts, bias=False)
        self.experts = nn.ModuleList([Expert(dim) for _ in range(num_experts)])

    def forward(self, x: torch.Tensor) -> Tuple[torch.Tensor, torch.Tensor, List[List[int]]]:
        # x: [batch, dim]
        logits = self.router(x)  # [batch, num_experts]
        probs = torch.softmax(logits, dim=-1)
        topk_vals, topk_idx = torch.topk(probs, self.top_k, dim=-1)
        # Normalize selected.
        topk_vals = topk_vals / topk_vals.sum(dim=-1, keepdim=True)
        out = torch.zeros_like(x)
        selected: List[List[int]] = []
        for b in range(x.shape[0]):
            idxs = topk_idx[b].tolist()
            selected.append(idxs)
            for rank, expert_idx in enumerate(idxs):
                out[b] = out[b] + topk_vals[b, rank] * self.experts[expert_idx](x[b:b + 1]).squeeze(0)
        return out, logits, selected

Expected outcome

Observed with the command above (exit 0). Selected stable fields from the end of the JSON:

  "active_parameters": 160,
  "total_parameters": 288,
  "top_k": 2,
  "num_experts": 4

The full output also contains the four output vectors, the four router-logit vectors and the two expert ids selected for each token. The active fraction is 160 / 288 = 0.556, so this tiny layer is not very sparse. The chapter’s larger Mixtral-shaped example is where the active fraction becomes small; the principle is the same, but the number is not transferable.

Mechanism and limitations

Documented: the layer rejects top_k > num_experts, computes router logits, softmaxes them, selects top-k, renormalises the selected weights and sums the selected expert outputs. run counts active parameters as the router plus top_k copies of one expert, and total parameters as the router plus every expert. The tests check the selected count, the active/total relationship, the output shape and JSON serialisation.

Observed: the command exited 0 with active_parameters: 160, total_parameters: 288, top_k: 2 and num_experts: 4. The targeted tests.test_ch19_moe is part of the canonical suite; no independent test count is claimed here.

Not shown: a trained router, real expert parallelism, communication cost, capacity limits, load balancing, or a production MoE model. The layer processes one token at a time in a Python loop; its latency is not a kernel benchmark. The active-parameter count is an accounting identity for this architecture, not a measurement of what the hardware read.

Challenge this result

Run with --top-k 1 and --top-k 4. Compute active parameters by hand for each before running. At top-k 4 every expert is active, so the layer should behave like a dense mixture; explain why the active count equals the total only in that case. Then change --experts 8 and compute the active fraction at top-k 2. Finally, explain why a lower active fraction does not automatically mean lower latency: routing, gather/scatter and load imbalance also cost time.

Source and verification

Canonical source: examples/ch19/moe.py at commit f958967d70992d7b09d9c75000bd8c543f1644d4. Tests: tests/test_ch19_moe.py . Chapter: Mixture-of-Experts Inference . The run above is a fresh local execution on Windows 11 with Python 3.13.15 and PyTorch 2.6.0+cu124; it uses a randomly initialised layer, not a trained model.

Understand this example

Copy this prompt into your AI tool. No code runs here.

Apply this example

Copy this prompt into your AI tool. No code runs here.