Mixture-of-Experts at Inference Time: How Routing Decisions Shape Cost and Quality

Why MoE Inference Behaves Differently Than You Expect

Mixture-of-Experts (MoE) is usually described as a training-time trick: split a giant feed-forward network into many smaller "experts" and train a router to send each token to only a few of them. That framing misses where the money is actually spent. Once a model ships, MoE inference cost and quality are decided at inference time, token by token, by a learned routing function you rarely control directly.

The contrast with dense models is sharp. In a dense transformer, every parameter participates in every token's forward pass, so per-token FLOPs scale linearly with model size. That makes dense cost predictable in shape — but not in magnitude: at small batch sizes, dense inference is also memory-bandwidth-bound, because you still stream every weight for every token.

In a sparse MoE, only a subset of experts activates per token, so a model with hundreds of billions of parameters can run with the per-token FLOPs of a far smaller one — on average.

That "on average" is the catch. Routing is simultaneously a cost lever and a quality lever. Push tokens toward fewer experts and you cut FLOPs, memory traffic, and latency — but you also change which computations produce the output. The coupling is not symmetric: routing decisions are set during training, and at inference you tune the serving system around those decisions, not the decisions themselves. The rest of this article unpacks how that coupling works and where the practical levers actually live.

The Anatomy of a Routing Decision

Every MoE layer contains a router — also called a gating network — that sits in front of the expert pool. For each token at each MoE layer, the router computes a score for every expert, applies a softmax, and produces a probability distribution over the pool. Those probabilities determine which experts fire and how their outputs are weighted.

The Router as a Learned Gatekeeper

The router is typically a small linear projection: it maps a token's hidden state to a vector of expert logits, then normalizes them. It is trained jointly with the rest of the model, so it learns to assign tokens based on patterns that reduce the training loss — not based on human-interpretable categories.

This matters because the router is a learned policy, not a fixed rule. There is no guarantee that "code tokens go to expert 3." The assignment emerges from data, initialization, and regularization, and it can shift during fine-tuning or drift under distribution change. (See: How fine-tuning reshapes MoE routing behavior)

Top-k Selection, Fine-Grained Experts, and the Sparsity Trade-off

After scoring, the model selects the top-k experts per token. The design space has shifted materially since the original Switch Transformer and GShard formulations, which used a small number of large experts with k=1 or k=2. Modern production MoEs — Mixtral 8x7B, DeepSeek-V2/V3, Qwen-MoE, and others — use fine-grained experts: many small experts with a higher k. DeepSeek-V3, for example, uses 256 routed experts per layer with top-8 selection, plus a small number of always-on shared experts that every token passes through.

Two consequences follow:

Cost scales with k × per-expert FLOPs, not with k alone. Fine-grained designs can raise k without proportional cost growth, because each activated expert is smaller. This is why "k=1 or k=2" is no longer the right default mental model.
Shared experts absorb common computation, freeing routed experts to specialize. They also stabilize routing, since every token gets a consistent baseline path regardless of which routed experts fire.

Routing happens per token and per layer, so decisions compound across depth — a token that routes poorly in layer 8 arrives at layer 9 with a distorted representation. The router's behavior is shaped during training by an auxiliary load-balancing loss that penalizes imbalanced expert utilization. This loss is a double-edged sword: too weak and the router collapses onto a few experts; too strong and the router flattens, producing near-uniform assignments that erode specialization and waste capacity.

[ILLUSTRATION: Diagram of a single MoE layer showing a token entering a router, producing a probability distribution over N experts, with top-k experts activated and their weighted outputs summed alongside a shared expert path.]

How Routing Directly Determines Inference Cost

Once you accept that routing drives cost, the next question is which cost. The intuitive answer — FLOPs — is usually wrong.

Active Parameters vs. Total Parameters

A 400B-parameter MoE with 40B active parameters per token performs roughly the same FLOPs per token as a 40B dense model. That is the headline efficiency claim, and it holds for compute-bound workloads. Vendor pricing and benchmark comparisons often lean on this number.

But two caveats matter. First, FLOPs equivalence is not quality equivalence: a well-trained 400B/40B MoE typically outperforms a 40B dense model of the same active count, because total capacity buys representational headroom even if it isn't all used per token. Second, active parameters describe only the math. The model still stores all 400B parameters, and at inference it must make the right experts available quickly. The gap between active and total parameters is where most operational surprises live. (See: MoE vs. dense model serving costs compared)

Prefill vs. Decode: Two Different Problems

The single most important operational fact about MoE serving is that prefill and decode stress different resources.

Prefill processes many tokens per forward pass. The router assigns them across experts, and the workload looks like a large, if irregular, GEMM. Prefill is usually compute-bound, and MoE's sparsity pays off cleanly.
Decode generates one token per sequence per step. With small batch sizes, each step touches only a handful of experts, but must stream their weights from memory. Decode is memory-bandwidth-bound, and the effective arithmetic intensity is low.

This asymmetry is why MoE throughput numbers quoted for prefill-heavy benchmarks (long-prompt summarization, RAG) often fail to predict decode-heavy workloads (chat, agent loops). Size your fleet against the decode case.

Memory Bandwidth, Not FLOPs, Is Usually the Bottleneck

In real serving systems, MoE decode is frequently memory-bandwidth-bound. Every expert's weights must be resident in VRAM or fetched from host memory or storage. If a token routes to an expert that isn't cached locally, you pay a transfer cost that dwarfs the arithmetic. The bandwidth hierarchy makes the cliff concrete: HBM on an H100 delivers roughly 3 TB/s per GPU, NVLink roughly 900 GB/s per GPU, PCIe Gen5 roughly 64 GB/s, and NVMe far less. An expert fetch that misses VRAM and lands on PCIe-attached host memory is ~50× slower than a VRAM hit.

This flips the usual optimization instinct. Adding compute rarely helps; reducing weight movement does. Practical consequences:

VRAM capacity determines how many experts stay resident, which caps how much sparsity you can actually exploit. Beyond that, you are offloading.
Interconnect bandwidth (NVLink, InfiniBand) sets the ceiling on cross-device expert access during all-to-all dispatch.
Batch size changes the aggregate routing pattern. The router still scores each token independently, but a larger batch draws from a wider slice of the expert pool, increasing the number of distinct experts touched per step and therefore aggregate memory pressure — even if per-token FLOPs are flat.
Quantization is a first-class lever. Because each expert is small and rarely all experts are hot simultaneously, MoE weights quantize well: FP8 and INT4 reduce both resident footprint and bytes moved, which directly attacks the bandwidth bottleneck. DeepSeek-V3 is trained and served in FP8 for exactly this reason.

Load Balancing and Expert Parallelism Overhead

Experts are typically sharded across GPUs (expert parallelism). Each MoE layer requires an all-to-all communication step: tokens are dispatched to the devices hosting their chosen experts, then results are gathered back. In fine-grained designs, this all-to-all happens more often and moves more metadata, which is a real cost that coarse-grained designs avoid.

If routing is imbalanced — say, 70% of tokens pick the same two experts — you get stragglers. Some GPUs idle while others saturate, and effective throughput collapses even though nominal FLOPs look fine. At the extreme, this becomes routing collapse: the router converges on a small subset of experts, the model behaves like a much smaller dense network, and you pay full memory cost for capacity you never use. Collapse is usually a training failure, but it shows up as an inference cost problem.

The capacity factor is the mechanism that bounds this during training: it caps how many tokens each expert will accept per batch, with overflow tokens dropped or rerouted. In production inference, most serving stacks raise or disable the capacity factor, because silently dropping tokens degrades output quality in ways that are hard to attribute. That means the load-balancing burden shifts back onto the router's training-time behavior and onto the serving system's placement and batching strategy.

Routing imbalance is the silent tax on MoE serving. A model can look efficient on paper and still leave a meaningful fraction of GPU cycles idle because a handful of experts became the default destination.

How Routing Shapes Output Quality

Cost is only half the story. Routing also determines what the model computes, and the quality implications are subtler than the efficiency ones.