Mixture of Experts: Big Model, Small Bill, Hidden Catch

When a lab says a new model has 100 billion parameters but “only 10 billion active”, that is mixture of experts. It is why some very large models run surprisingly fast, and also why running them locally is harder than the “active” number suggests.

How a dense layer differs from an MoE layer

In a standard transformer block, each token goes through attention and then a single feed-forward network (FFN). Every token uses every weight in that FFN.

In an MoE block, the single FFN is replaced by many parallel FFNs, the experts, and a small router:

token vector
   |
 [router]  ->  scores for each expert
   |
 pick top-k experts (k is small, often 1 or 2)
   |
 expert_i(token), expert_j(token)  ->  combine weighted by router scores

Attention layers are typically still shared across all tokens. Only the feed-forward part is sparse. The Switch Transformer work pushed this to an extreme by routing each token to a single expert, showing that capacity can scale a lot without scaling per-token compute.

Total versus active parameters

A model with 64 experts per layer and top-2 routing uses roughly 2/64 of the expert weights per token. That is where “big model, small bill” comes from.

What makes MoE hard in practice

When MoE is worth it

• Serving at scale with ample memory: high throughput per dollar, because each token touches fewer weights.
• Wanting more knowledge capacity without proportionally more compute at each step.

And when it is not:

• Memory-constrained deployment, such as one consumer GPU, where total parameters decide whether it fits at all.
• Small batch, latency-critical use, where expert routing overhead and underused experts erode the savings.

Takeaway

Mixture of experts decouples a model’s capacity from its per-token compute by activating only a few specialists at a time. It is a real efficiency win with a real cost: the full parameter set must live in memory, and routing makes training and serving harder. Always ask for both the total and the active parameter count.