Serving Mixture-of-Experts Models in Production: The Cost Math Nobody Shows You

The sticker-price trap

Every Mixture-of-Experts (MoE) model launch follows the same script. The vendor highlights a huge total parameter count, then pivots to a small active parameter count to imply you get frontier quality at a fraction of the serving cost. The implication is seductive: if only 20B of 600B parameters fire per token, your inference bill should look like a 20B dense model.

It won't. Your production bill is set by total weights that must reside in memory, the bandwidth to move them, and how fully you utilize the GPUs you were forced to buy — not by the active-parameter number on the slide. The active count governs compute per token; everything else governs cost per token, and the two diverge sharply for MoE.

This article walks the full MoE serving cost equation line by line: weight memory, KV cache, expert-parallel communication, routing imbalance, utilization, and capacity headroom. By the end you'll be able to estimate cost per million tokens for a Mixture-of-Experts deployment before you sign a contract — and spot the assumptions vendor decks quietly leave out.

Why MoE Serving Economics Break the Dense-Model Playbook

Active vs. total parameters: the number vendors quote

A sparse MoE layer replaces one feed-forward network with many "experts" and a router that sends each token to a small subset — typically top-2 of 8, 64, or 256. With 64 experts and top-2 routing, roughly 3% of expert weights activate per token. That sparsity is real, and it's why MoE delivers more quality per FLOP than a dense model of equal active size.

Two caveats the vendor slide omits. First, "3% of expert weights" is not "3% of model weights": attention layers, embeddings, the router, and any shared experts are always active. The active-parameter figure (e.g., 20B of 600B) already folds these in — but the "3%" shorthand does not, and conflating the two inflates the apparent savings. Second, sparsity is a compute property, not a memory property. Every expert must be resident and ready, because the router can send any token to any expert. The dense-model intuition — "FLOPs drive cost" — assumes weights stream from memory in proportion to compute. For MoE, all weights occupy memory whether or not they fire on a given token. (See also: how large language model inference actually works.)

Memory bandwidth, not FLOPs, sets your bill

Inference is overwhelmingly memory-bound at realistic batch sizes. Each generated token requires reading the weights that participate in the forward pass. For a dense model, bytes-read scales with parameter count. For MoE, the bytes-read per token scale with active parameters — but the bytes that must be stored and addressable scale with total parameters.

That gap is the whole story. You pay to provision memory capacity for the total model, then pay again in bandwidth and communication to shuffle activations among experts scattered across GPUs. A dense model of the same active size needs a fraction of the hardware footprint and none of the all-to-all traffic. The MoE premium is a memory-and-interconnect premium, and it is largely fixed.

One nuance worth flagging: at low batch sizes, bytes-read per token really does track active parameters, because each token touches only its routed experts. At production batch sizes, each GPU reads the weights of all its local experts for the tokens routed to it, and effective bytes-read per token can drift above the naive active-param estimate. The memory-capacity floor, however, never moves.

The Cost Math, Line by Line

Step 1 — Weight memory and minimum GPU count

Start with the floor no traffic pattern can go below:

Weight memory = total parameters × bytes per parameter + overhead

At FP8 (1 byte/param) a 600B-parameter MoE needs ~600 GB just for weights. Add KV cache headroom, activations, and framework overhead and you're realistically at 700–750 GB per replica. On 80 GB GPUs that's 700 ÷ 80 ≈ 8.75, i.e., a minimum of 9–10 GPUs per replica — and because expert parallelism needs power-of-two or topology-aligned groups, you'll likely land on 16.

Note the hardware dependency: 80 GB is H100/A100-80GB class. On H200 (141 GB) the same weights fit in 5 GPUs, and the replica shape changes entirely. Run this arithmetic against the specific SKU you're being quoted.

Here's the trap: that 16-GPU replica costs the same whether it serves 5 requests per second or 500. Your minimum GPU count is set by model size, not load. A dense 20B model fits on a single 80 GB GPU and scales down to one. The MoE replica is a fixed monthly line item before a single token is generated. (Related: GPU sizing and capacity planning for LLM workloads.)

[ILLUSTRATION: Bar chart comparing weight memory footprint of a 600B-total/20B-active MoE versus a 20B dense model, showing the MoE bar extending across 8+ GPUs while the dense bar fits in one.]

Step 2 — KV cache and concurrency ceiling

KV cache scales with context length, batch size, and the model's attention configuration — not with sparsity. For each concurrent request you reserve:

KV bytes ≈ 2 × layers × KV heads × head dim × sequence length × bytes per element

Worked example: a 60-layer model with grouped-query attention (8 KV heads, 128 head dim), 8K context, FP16 cache:

2 × 60 × 8 × 128 × 8192 × 2 ≈ 2 GB per request.

The critical interaction: expert weights already consumed most of your VRAM, so the KV cache gets the leftovers. On a tightly provisioned MoE replica you may have only 10–15 GB per GPU for cache. At ~2 GB per 8K request that's roughly 5–7 concurrent sequences per GPU — a few dozen per replica. Models with MQA or fewer KV heads do better; models with more layers or longer context do worse.

Low concurrency ceiling means low batch size, which means low GPU utilization, which means high cost per token. The memory you spent on dormant experts is memory you couldn't spend on throughput. (See: KV cache optimization techniques for high-throughput serving.)

Step 3 — Expert-parallel communication overhead

Because experts are sharded across GPUs, every MoE layer triggers an all-to-all exchange: route tokens to the GPUs holding their experts, compute, route results back. This happens at every MoE layer, every token, every forward pass.

The cost depends entirely on your interconnect tier:

NVLink within a node — ~900 GB/s bidirectional per GPU on H100 (NVLink 4); all-to-all overhead is modest, often under 10% of layer time.
PCIe within a node — ~64 GB/s bidirectional on Gen5 x16, roughly 14x slower than NVLink; all-to-all can dominate and force larger batch sizes to amortize.
Cross-node (InfiniBand/Ethernet) — highest latency and cost; you pay for fat network fabric and still lose throughput to communication stalls.

The rule: keep expert groups inside NVLink domains. A model that technically fits across two nodes at PCIe speeds may deliver roughly half the throughput of the same model packed into one NVLink node — which can approach a 2x effective cost per token. Treat that factor as workload-dependent, not a constant; measure it on your traffic before committing.

Step 4 — Routing imbalance and capacity factor

The router is not a fair scheduler. In practice some experts get hot and some stay cold, and the imbalance is systematic, not random. Serving frameworks handle this with a capacity factor — a per-expert token buffer sized as a multiple of the average load. Set it too low and tokens get dropped (quality degrades silently); set it too high and you reserve memory and compute for tokens that never arrive.

Two consequences for cost:

Effective utilization of expert weights is below 100% even when the GPU is "busy," because hot experts bottleneck while cold experts idle.
Expert replication (placing a copy of hot experts on multiple GPUs) trades memory for balance. It raises your memory floor — the same floor from Step 1 — to buy back throughput.

Different serving stacks implement this differently: vLLM, TensorRT-LLM, SGLang, and DeepSpeed-MoE each have their own EP strategies, capacity-factor defaults, and replication support. The cost model you build should assume the framework's actual behavior, not the theoretical ideal.

Step 5 — Utilization and the fixed-cost denominator

Put Steps 1–4 together and the cost-per-token equation looks like this:

Cost per million tokens = (fixed GPU cost per replica × replicas) ÷ (tokens per second per replica × seconds per month) × 1,000,000

The numerator is dominated by Step 1 — the minimum replica count — and barely moves with load. The denominator is where MoE quietly loses: communication stalls (Step 3), routing imbalance (Step 4), and a tight KV ceiling (Step 2) all push tokens-per-second down. Every point of lost throughput multiplies the cost per token.

Step 6 — A worked example

Assume a 600B-total / 20B-active MoE at FP8, served on 16× H100 80 GB at an on-demand list price of ~$3/GPU-hour (rented, not owned; owned hardware amortizes differently but the shape of the math is identical).

Fixed cost per replica: 16 × $3 × 730 hours ≈ $35,000/month.
Throughput: assume 1,500 output tokens/sec/replica after communication and routing losses (measure this; it is the single most important number and the one vendor decks never give you).
Monthly token capacity: 1,500 × 2,592,000 sec ≈ 3.9B tokens.
Cost per million output tokens: $35,000 ÷ 3,900 ≈ ~$9/M tokens — if you run at full utilization.

Now apply realistic utilization. At 40% average utilization the effective cost is ~$22/M tokens. At 15% — common for a low-traffic internal deployment pinned to a fixed replica — it's ~$60/M tokens. The same hardware, the same model, a 6x swing in unit cost driven entirely by load and utilization.

Compare a dense 20B on a single GPU at ~$3 × 730 ≈ $2,200/month at 1,000 tokens/sec, ~2.6B tokens/month, ~$0.85/M tokens at full utilization. The MoE is not 10x more expensive per token at saturation — but at low utilization the gap widens dramatically, and that is exactly the regime most B2B deployments live in.

What to Demand Before You Sign

Total weights, bytes per parameter, and the actual serving precision (FP8 vs. FP16 changes the memory floor by 2x).
Minimum replica shape and GPU SKU — and whether the vendor's number assumes NVLink-local expert groups.
Measured tokens/sec/replica at your target context length and batch size, not a peak benchmark.
Capacity factor and token-drop behavior under skewed routing.
Utilization assumption baked into any quoted cost per token — and what happens to your bill if your traffic is half the forecast.

The active-parameter number is a compute spec. Your invoice is a memory-and-utilization spec. Price the second one.