12.2 Mixture of experts and sparse models
You can explain why a large model can be cheap to run.
Before:08. Large Language Models
Mixture-of-experts models activate only a fraction of their parameters per token — a router picks experts — which is how modern large models stay affordable to run, and why active-versus-total parameter counts explain otherwise baffling pricing. It sits in the frontier module as the architecture behind current flagships. The sizing trap is infrastructure planned from active parameters alone: all the experts must live in memory, whether or not this token used them.
Work through these
Router networks and expert selection
A small network chooses which specialist components handle each input, so only part of the model runs. That routing is the whole idea.
Active vs. total parameters
The distinction between how many parameters exist and how many are used per input is what lets a very large model be affordable to run. It also makes parameter counts less comparable than they look.
Load balancing and training instabilities
Routers tend to favour a few components, which wastes the rest and destabilises training. Keeping the load spread is an active part of training these models.
Serving implications of MoE
All the components must be in memory even though only some run, which changes the hardware requirement substantially. This is why the saving is in computation rather than in memory.
Sign in to keep your progress.
Free resources
We haven't checked most of these for screen reader use yet.
Links last checked 29 Aug 2026.
Stuck here?
Ask a mentor. A real person answers, and they can see exactly which topic you're on. Usually within a couple of working days.
Checking your session…
Topics shown in module order.