12.2 Mixture of experts and sparse models

What this is and why it exists

A very large model that is cheap to run sounds like a contradiction, and the resolution is that only a fraction of it runs for any given input. A small network chooses which specialist components handle each token; the rest sit idle. That one idea explains parameter counts that no longer compare across models, and it explains a memory requirement that catches people out — because every component must be in memory even though few of them fire.

The vocabulary

• Expert — one of several parallel sub-networks within a layer.
• Router — the small network choosing which experts handle each token.
• Top-k routing — sending each token to its k highest-scoring experts.
• Active parameters — those used for one token.
• Total parameters — all that exist.
• Load balancing — keeping work spread across experts rather than concentrated.
• Auxiliary loss — an extra training term encouraging that spread.
• Expert parallelism — placing different experts on different devices.

The mental model

Replace one feed-forward block with several, and add a chooser. Where a dense layer applies the same network to every token, this arrangement holds a number of parallel networks — the experts — and a small router that scores them for each token and sends it to the best one or two. Only those run. The rest contribute nothing for that token and cost nothing in computation.

That is the whole idea, and everything else in the topic is a consequence. The layer has the parameters of all the experts and the computation of one or two, so capacity and cost are decoupled — you can add capacity by adding experts without adding computation per token, which is precisely the thing dense scaling cannot do.

Which is why parameter counts stopped being comparable. A dense model's parameter count is both its capacity and its cost. A sparse model has two numbers: total parameters, which is all of them, and active parameters, the fraction used per token, which is what the computation and much of the speed follow. A model with a very large total and a modest active count behaves computationally like a much smaller model and holds the knowledge of a much larger one — and that is the explanation for pricing and speed that otherwise look baffling when set against a headline parameter count.

So when you read a size, ask which number it is. Comparing one model's total against another's active is comparing two different quantities, and it is done constantly.

Load balancing is an active part of training, not a detail. Routers left alone collapse: a few experts get chosen slightly more often early, so they train more, so they get better, so they get chosen more still. The result is a handful of overworked experts and a large number that learned nothing — capacity paid for and unused.

The standard remedy is an auxiliary loss that penalises uneven assignment, pushing the router towards spreading tokens across experts. There are also capacity limits per expert, where a token routed to a full expert is dropped or sent elsewhere, which bounds the imbalance mechanically.

Two training instabilities are characteristic, and knowing them explains why these models were difficult before they were routine. The routing decision is discrete — this expert, not that one — which does not differentiate cleanly, so training has to work around it. And the balancing pressure fights the routing objective: the router wants the best expert and the auxiliary loss wants an even spread, and weighting the two badly gives either collapse or a router that assigns almost randomly.

Then the serving consequence, which is the trap. All experts must be in memory even though few run per token. Any token might be routed anywhere, so every expert's weights must be resident and reachable — which means memory requirements follow the total parameter count, and computation follows the active count.

Size infrastructure from active parameters alone and the model will not load. That is the whole warning, and it is a common and expensive mistake, because the model's advertised behaviour is its computational lightness and its memory demand is the other number.

The rest of the serving picture follows from that. Since the memory is large, these models are frequently split across devices with different experts on different devices — which means routing a token can require sending it to another device, so the interconnect between devices becomes part of the inference path and the communication cost from the distributed-training topic reappears at serving time. Batching also behaves differently: a batch of tokens routed to many different experts activates most of the model, so the computational saving is smaller in a large mixed batch than the active-parameter figure suggests.

And the memory-bound picture from the accelerator topic applies with force. Generation reads weights to produce one token, and here it reads a different subset per token, so the access pattern is scattered as well as large. The saving is in computation rather than in memory, which is exactly the sentence to remember, and whether that is the saving you needed depends on which resource was binding — for a compute-limited setting it is transformative, and for a memory-limited one it buys much less than the headline suggests.

The practical summary for somebody choosing or deploying one. Read both parameter numbers and know which each figure refers to. Size memory from the total and estimate computation from the active count. Expect the advantage to be largest at low batch sizes and to shrink as batches mix. And remember the perishability of everything specific here: the architecture idea is stable and the particular arrangements are not, so the numbers belong in the topic's resources rather than in your head.

What you should now be able to explain or do

Describe the router-and-experts arrangement and say what it decouples. Distinguish active from total parameters and use the distinction to read a size or a price sensibly. Explain why routers collapse and what an auxiliary loss and capacity limits do about it. Name the two characteristic training instabilities. State the serving rule — memory tracks total, computation tracks active — and size infrastructure accordingly. Say why the interconnect enters the inference path and why large mixed batches reduce the saving. Say which resource this helps and which it does not.

Check yourself