Mixture of experts
Expert imbalance is a serving bug, not a training curiosity
A mixture-of-experts layer is a router plus a set of smaller networks. At serve time the router’s skew becomes idle GPUs, dropped tokens, or both.
Training papers talk about load-balancing losses because a few experts can absorb most of the gradient. Serving inherits the same skew and adds a deadline. The step cannot finish until the busiest expert finishes. Unused experts still occupy memory. The average expert is not the one on the critical path.
Capacity is a hard cap
Each token is sent to the top k experts. If an expert is assigned more tokens than its capacity, the implementation either drops the overflow or expands the step to fit it. Capacity is often written as:
capacity = capacity_factor × tokens_per_step × k / num_experts
A capacity factor of 1.0 is the even-split budget. It is not a comfortable operating point. Real routers are peaked, so a factor near 1 drops tokens on the hot experts and leaves the cold ones padded. Raising the factor keeps more tokens and pads more slots. Dropped tokens change the model’s computation. Padded slots change the latency. Neither shows up in a parameter count.
Decode is the worse case
Prefill can group thousands of tokens from one prompt, so a single step has enough samples for the router to look closer to its average. Decode, especially at small batch, sends a handful of tokens per step. With a handful of draws, one expert can legally receive all of them. The all-to-all that follows still synchronizes every device. The step time tracks the max, then everyone waits.
That is why an MoE model can look efficient in a prefill-heavy benchmark and miss its decode SLO in chat. The communication pattern did not change. The number of tokens available to balance the router did.
What to measure on the live route
- tokens dispatched per expert, as a distribution, not a mean
- fraction of tokens dropped, split by layer
- padding fraction inside the expert GEMM
- all-to-all time as a share of the MoE block
- decode latency at the batch you will actually run, including batch 1
If one expert is hot on your domain, a global load-balancing loss from pretraining will not fix the server. The practical responses are local: a higher capacity factor where drops are unacceptable, expert parallelism that matches the hot set, or a route that refuses batch sizes too small to fill the router. Pick from the trace. The model card will not contain it.