What Is a Mixture of Experts (MoE)?

The problem MoE solves

A standard ("dense") transformer uses every one of its parameters on every single token it processes. That's simple, but it means making a model bigger — to give it more knowledge and capability — also makes every single request proportionally more expensive to run, since all of that bigger model has to activate every time.

The MoE idea

A Mixture of Experts model splits parts of the network (usually the feed-forward layers) into many smaller specialized sub-networks called "experts." For any given token, a small lightweight router decides which handful of experts (often just 2 out of dozens) actually get used. The model's total parameter count can be enormous, but only a fraction of it is active for any single token — giving you much of the capacity benefit of a huge dense model at closer to the inference cost of a much smaller one.

Why providers use it

MoE has become common in frontier models precisely because it decouples two things that used to move together: total knowledge/capacity, and per-request compute cost. A well-designed MoE model can match or beat a dense model with far more active parameters, at meaningfully lower serving cost — which matters enormously at the scale flagship providers operate at.

The trade-off

MoE isn't free complexity-wise: training the router well (so it reliably sends each token to the experts actually suited for it) is genuinely hard, and MoE models typically need more total memory (to hold every expert, even the unused ones) even though they need less compute per token. It's a trade of memory footprint for compute efficiency, not a strict win on every dimension.