What is a mixture-of-experts (MoE) model?

A mixture-of-experts model replaces the dense feed-forward block in each transformer layer with N independent feed-forward "experts" plus a small router (gating network). For each token the router scores all N experts and sends the token to only its top-k of them — typically 1 or 2 — and the layer's output is the router-weighted sum of just those experts' outputs. The rest of the layer, notably attention, still runs densely for every token. The point is to decouple the parameters a model stores from the parameters it computes per token: N experts means N times the feed-forward parameters in memory, but each token only pays for k of them. Mixtral 8x7B is the canonical open example — 8 experts, top-2 routing.

How does MoE save compute? Does it save memory?

It saves compute (FLOPs), not memory. The FLOPs a token costs are set by its active parameters — the shared attention plus the k experts it was routed to — so a sparse model runs at the compute of a much smaller dense model. But every expert has to be resident in memory to be routable, so the model still needs VRAM for its full total parameter count. Mixtral 8x7B holds about 46.7B parameters but activates only ~13B per token: roughly a 3.6x gap between stored and computed. Answering "MoE saves compute because it has fewer parameters" is the classic mistake — it has more parameters, it just uses fewer of them at a time.

What is the load-balancing auxiliary loss in MoE?

Left alone, a router collapses: a few experts get slightly more traffic, train faster, and attract even more, until most experts are dead weight. The language-modeling objective does nothing to prevent this, so MoEs add an auxiliary load-balancing loss. The Switch Transformer's version is loss = α · N · Σ fᵢ·Pᵢ, where fᵢ is the fraction of tokens routed to expert i, Pᵢ is the mean router probability on expert i, N is the number of experts, and α is a small coefficient (10⁻²). It is minimized when both are uniform (every expert gets 1/N of the load), so its gradient pushes the router back toward balance. It is a soft push, not a hard guarantee — real MoEs still show imbalance, which is why newer models add device-level terms or chase auxiliary-loss-free balancing.

What is capacity factor and why are tokens dropped in MoE?

Because experts run as fixed-shape batched matmuls, each expert is allotted a fixed number of token slots per batch — its capacity — set by expert capacity = (tokens per batch / number of experts) × capacity factor. A capacity factor of 1.0 gives each expert its fair share; higher values buy buffer for uneven routing at the cost of padded, wasted compute. When more tokens route to an expert than it has slots, the overflow tokens are dropped: their computation is skipped and their representation passes unchanged to the next layer through the residual connection. In top-2 routing a token is only fully dropped when both of its chosen experts are over capacity. Dropping is tolerated as noise during training; at inference you can raise the capacity factor high enough to avoid it.

What are fine-grained and shared experts (DeepSeekMoE)?

They are two refinements introduced by DeepSeekMoE to make experts specialize better. Fine-grained segmentation splits each expert into several thinner ones and routes to proportionally more of them, so there are far more ways to combine specialists at the same compute cost. Shared expert isolation reserves a small number of experts that are always active for every token, to absorb common, general-purpose computation so the routed experts don't each have to relearn it. DeepSeekMoE-16B uses 2 shared plus 64 fine-grained routed experts (activating 6 routed per token): 16.4B total, 2.8B active. DeepSeek-V2 scales the recipe to 236B total / 21B active with 2 shared and 160 routed experts, top-6. Fine-grained-plus-shared is now the dominant open MoE design.

Mixture of Experts, Routed Honestly

A dense model spends every parameter on every token; a mixture-of-experts model keeps far more parameters but touches only a slice per token — huge in memory, cheap in compute at once. The whole design turns on one question: which experts does a token go to? We compute the routing decision live — a real softmax gate over 8 experts, top-k selection, a capacity cap that drops overflow tokens, and the load-balancing loss that fights router collapse — then count Mixtral 8x7B down from 47B parameters to the 13B it actually uses, cover expert parallelism and its all-to-all bill, and the DeepSeek fine-grained-plus-shared recipe.

Concept · AI / ML. The source ↗

A free, interactive, animated visual explainer of Mixture of Experts, Routed Honestly — built to be understood, not skimmed.

Questions

What is a mixture-of-experts (MoE) model?: A mixture-of-experts model replaces the dense feed-forward block in each transformer layer with N independent feed-forward "experts" plus a small router (gating network). For each token the router scores all N experts and sends the token to only its top-k of them — typically 1 or 2 — and the layer's output is the router-weighted sum of just those experts' outputs. The rest of the layer, notably attention, still runs densely for every token. The point is to decouple the parameters a model stores from the parameters it computes per token: N experts means N times the feed-forward parameters in memory, but each token only pays for k of them. Mixtral 8x7B is the canonical open example — 8 experts, top-2 routing.
How does MoE save compute? Does it save memory?: It saves compute (FLOPs), not memory. The FLOPs a token costs are set by its active parameters — the shared attention plus the k experts it was routed to — so a sparse model runs at the compute of a much smaller dense model. But every expert has to be resident in memory to be routable, so the model still needs VRAM for its full total parameter count. Mixtral 8x7B holds about 46.7B parameters but activates only ~13B per token: roughly a 3.6x gap between stored and computed. Answering "MoE saves compute because it has fewer parameters" is the classic mistake — it has more parameters, it just uses fewer of them at a time.
What is the load-balancing auxiliary loss in MoE?: Left alone, a router collapses: a few experts get slightly more traffic, train faster, and attract even more, until most experts are dead weight. The language-modeling objective does nothing to prevent this, so MoEs add an auxiliary load-balancing loss. The Switch Transformer's version is loss = α · N · Σ fᵢ·Pᵢ, where fᵢ is the fraction of tokens routed to expert i, Pᵢ is the mean router probability on expert i, N is the number of experts, and α is a small coefficient (10⁻²). It is minimized when both are uniform (every expert gets 1/N of the load), so its gradient pushes the router back toward balance. It is a soft push, not a hard guarantee — real MoEs still show imbalance, which is why newer models add device-level terms or chase auxiliary-loss-free balancing.
What is capacity factor and why are tokens dropped in MoE?: Because experts run as fixed-shape batched matmuls, each expert is allotted a fixed number of token slots per batch — its capacity — set by expert capacity = (tokens per batch / number of experts) × capacity factor. A capacity factor of 1.0 gives each expert its fair share; higher values buy buffer for uneven routing at the cost of padded, wasted compute. When more tokens route to an expert than it has slots, the overflow tokens are dropped: their computation is skipped and their representation passes unchanged to the next layer through the residual connection. In top-2 routing a token is only fully dropped when both of its chosen experts are over capacity. Dropping is tolerated as noise during training; at inference you can raise the capacity factor high enough to avoid it.
What are fine-grained and shared experts (DeepSeekMoE)?: They are two refinements introduced by DeepSeekMoE to make experts specialize better. Fine-grained segmentation splits each expert into several thinner ones and routes to proportionally more of them, so there are far more ways to combine specialists at the same compute cost. Shared expert isolation reserves a small number of experts that are always active for every token, to absorb common, general-purpose computation so the routed experts don't each have to relearn it. DeepSeekMoE-16B uses 2 shared plus 64 fine-grained routed experts (activating 6 routed per token): 16.4B total, 2.8B active. DeepSeek-V2 scales the recipe to 236B total / 21B active with 2 shared and 160 routed experts, top-6. Fine-grained-plus-shared is now the dominant open MoE design.

Mixture of Experts, Routed Honestly

Questions

Related explainers