Deep Learning Inference in 2026: From Attention to Sparse MoE — What Enterprise Teams Need to Know

Three forces that set your cost-per-token

Content per outline: decode vs prefill, definitions, three forces, hardware is third variable.

Draft:

"Token generation has two phases with opposite bottlenecks. Prefill processes your entire prompt in one parallel pass and is compute-bound — limited by floating-point operations (FLOPs), the raw arithmetic the GPU can perform. Decode generates output one token at a time and is memory-bandwidth-bound — limited by how fast the GPU moves data, not how fast it calculates. Every architectural decision below maps to one of these two bottlenecks.

Three forces set your cost-per-token:

1. KV cache memory — the per-user attention history that grows linearly with context length and caps how many users fit on one GPU.
2. Sparse activation — mixture-of-experts (MoE) routing that cuts per-token compute while total weights stay large.
3. Serving software — batching, caching, and scheduling that decide how much of the hardware you actually use.

Hardware matters, but it is the third variable, not the first. Two teams on identical GPUs routinely see 2-4× different cost-per-token purely from the decisions in this list."

What changed since 2024

Per outline: dense defaults, three breaks, MoE-first portfolios.

Draft:

"In 2024, the default enterprise serving target was a dense model: every parameter activates for every token. Llama 3 70B on H100s, quantized to FP8 or INT4, was the reference architecture, and capacity planning meant counting parameters.

Three things broke that default:

1. Context lengths grew from 8K to 128K and beyond, multiplying cache footprint 16-fold and turning KV memory into the binding constraint.
2. DeepSeek-V3 shipped in December 2024 and DeepSeek-R1 followed in January 2025 — open-weight sparse MoEs with 671B total parameters but only 37B active per token, matching dense frontier quality at a fraction of the serving compute.
3. Open-weight MoE became a portfolio category, not a curiosity: Qwen3-235B-A22B (235B total, 22B active) and Llama 4 Maverick (~400B total, 17B active) gave teams credible sparse options at every size point.

By 2026, platform teams run MoE-first portfolios: a sparse model as the default workhorse, with dense checkpoints retained for fine-tuning-heavy, latency-critical, or memory-constrained workloads. The rest of this article is the math behind that portfolio decision."

Attention Mechanics That Move the P&L

Intro line: "Attention is where memory economics live. Three mechanisms decide your cache bill: the KV cache itself, the attention variant you choose, and how you handle long context."

KV cache math: run this formula before you buy GPUs

"The KV cache stores Key and Value tensors for every token in the context, so attention can be computed without reprocessing earlier tokens. Every token you keep in context holds a reservation of GPU memory for the life of the request. The formula:

KV cache per token = 2 × layers × kv_heads × head_dim × bytes_per_parameter

The 2 covers Key and Value. Worked example for a Llama-3-70B-class model in FP16 (80 layers, 8 KV heads, head dimension 128, 2 bytes):

2 × 80 × 8 × 128 × 2 = 327,680 bytes ≈ 320 KB per token

At 128K context (131,072 tokens), one sequence holds:

327,680 × 131,072 ≈ 42.9 GB

Callout: At 128K context, a single sequence's KV cache is ~42 GB in FP16 for a Llama-3-70B-class model. Run the numbers before you buy: with INT4 weights (~35 GB) one such sequence fits on an 80 GB H100; with FP8 weights (~70 GB) it doesn't. Either way, you are serving roughly one long-context user per GPU — not a fleet of concurrent users.

Two consequences follow. Grouped-query attention (GQA) is why the number is survivable at all: Llama 3 70B shares 8 KV heads across 64 query heads, an 8× cache cut that modern dense models treat as table stakes. And the metric that matters for capacity is not cache per token but concurrent users per GPU — cache per token × context × users, plus weights, must fit in VRAM. Teams that run this math first buy fewer GPUs."

GQA vs MQA vs MLA: the attention-variant tradeoff

Definitions: "Attention variants differ in how many Key/Value heads they keep, and therefore how much cache each token costs.

• Multi-head attention (MHA) — every query head has its own K/V head. Best quality, largest cache.
• Grouped-query attention (GQA) — query heads share KV heads in small groups (8 KV heads for 64 query heads in Llama 3 70B). Near-MHA quality at roughly 8× less cache than MHA.
• Multi-query attention (MQA) — all query heads share one KV head. Maximum cache savings (64× vs MHA here), but the largest measured quality drop.
• Multi-head latent attention (MLA) — DeepSeek's approach: K/V are projected into a low-rank latent vector and recomputed on the fly, cutting cache ~93% versus MHA while matching or beating MHA quality in DeepSeek's ablations.

Variant	KV heads (70B-class)	Cache vs MHA	Quality vs MHA	Found in
MHA	64	1×	baseline	older/large models
GQA	8	~8× less	near-baseline	Llama 3, Mistral, Qwen dense
MQA	1	~64× less	measurable drop	PaLM, early Falcon
MLA	latent (576 dims/token/layer)	~93% less	matches or beats MHA	DeepSeek-V2/V3, R1

The industry converged on GQA for dense models and MLA for the DeepSeek line because the quality tax of extreme sharing is real. MQA's single shared head measurably degrades downstream quality in the original ablations; GQA recovers most of it; MLA sidesteps the tradeoff by compressing rather than sharing. When you evaluate a model, the attention variant is a proxy for both its cache bill and its quality ceiling."

Wait — need to double check "PaLM uses MQA" — yes, PaLM used MQA. Falcon used MQA (Falcon 40B multi-query? Falcon used multi-query attention for 40B, multi-head for 7B). Fine.

MLA latent: 512 (kv_lora_rank) + 64 (qk_rope) = 576 per token per layer. Cache vs MHA ~93.3% reduction per DeepSeek-V2 paper. Good.

Long context: the O(n²) tax and how 2025 architectures dodged it

"Prefill compute scales with the square of context length: attention compares every token against every other token, so doubling context roughly quadruples prefill FLOPs. Long context is not just a memory problem; it is a latency problem. Time-to-first-token (TTFT) — how long a user waits before generation starts — is dominated by prefill, and at 128K tokens it can reach tens of seconds on a single GPU without mitigation.

Four mitigations now ship as standard practice:

1. Sliding-window and hybrid attention. Many local layers attend to a fixed window; a few global layers attend everywhere. Hybrids such as Gemma 3 and Llama 4 (whose iRoPE interleaves local and global layers) cut attention cost dramatically while keeping most long-range capability.
2. FlashAttention-3. The Hopper-optimized kernel overlaps compute with memory movement and supports FP8, running up to ~2× faster than FlashAttention-2 on H100 at FP16 — and more with FP8.
3. Chunked prefill. Splitting the prompt into chunks and interleaving them with decode steps keeps GPUs busy and stops one giant prompt from stalling every other user.
4. Prefill/decode (P/D) disaggregation. Dedicated prefill and decode pools, each sized for its own bottleneck, at the cost of moving KV caches between them.

If your product promises long-context agents, TTFT is the metric your users feel first — and the one most capacity plans ignore.

Enjoying this breakdown? We publish one deep dive like this every week on inference economics. Subscribe to get the next one — on quantization tradeoffs — in your inbox."

Hmm, the outline says CTA after TTFT pain point. Good. Keep the CTA as blockquote.

Sparse MoE Explained

How routing works

"A sparse mixture-of-experts model replaces one giant feed-forward block with many smaller expert blocks plus a router. For each token, the router scores all experts and activates only the top-k. DeepSeek-V3, for example, holds 256 routed experts plus 1 shared expert and activates 8 routed experts per token; Qwen3-235B-A22B activates 8 of 128 experts; Llama 4 Maverick activates 1 of 128 per token (plus a shared expert).

The result is a decoupling that dense models cannot offer: total parameters set the model's knowledge capacity, while active parameters set the per-token compute bill. DeepSeek-V3 carries 671B parameters but pays for roughly 37B per token. During training that capacity is a quality win; during serving it is a cost lever.

Two refinements matter for operators. First, load balancing is a first-class engineering problem: early MoEs needed auxiliary balancing losses that traded away quality, while DeepSeek-V3 introduced an auxiliary-loss-free scheme that adjusts per-expert bias to keep traffic even. Second, a shared expert always runs, capturing common patterns and letting routed experts specialize."

Llama 4 Maverick: uses 1 shared expert + routed; each token activates... Maverick: 128 routed experts, 1 activated per token? Llama 4 Maverick: "128 experts, each token routed to a single expert" — I believe Maverick activates 1 expert per token plus shared. Scout: 16 experts, 1 active. Yes, Maverick: interleaved MoE layers with 128 experts and top-1 routing + shared expert. I'll say "activates 1 routed expert per token alongside a shared expert." Safe.

MoE vs dense: the tradeoff table

Dimension	Dense (Llama 3 70B)	Sparse MoE (DeepSeek-V3 class)
Params	70B total, 70B active	671B total, 37B active
Per-token FLOPs	high	~10-20× lower
Weights in memory (FP8)	~70 GB	~671 GB
Decode at low batch	bandwidth-bound on 70B	bandwidth-bound on 37B — faster
Prefill throughput	good	excellent (compute cut)
Serving complexity	low	high (routing, expert parallelism)
Fine-tuning	simple, well-tooled	harder; expert imbalance risks

Analysis paragraph: "MoE shifts cost from compute to memory and coordination. You trade a bigger memory footprint and more complex serving for a fraction of the per-token arithmetic. At low concurrency, decode reads only the selected experts' weights, so memory traffic scales with active parameters and MoE decodes faster. At high concurrency, the batch touches every expert anyway, so weight reads amortize across many tokens and the remaining win is compute — which is exactly where prefill-bound, long-context workloads live."

Wait, at high batch: per step, all experts' weights get read (traffic ∝ total params), but amortized per token it's total_params/batch_size — which can be lower than active_params if batch > total/active ratio (671/37 ≈ 18). So at very high batch, per-token weight traffic can actually be lower than dense! But cache traffic also grows. The simplified honest version: "the win narrows to compute and cache" — hmm, my earlier note said the win narrows. Let me phrase carefully: "At high concurrency a large batch touches every expert in a single step, so weight reads scale with total parameters — but amortized across hundreds of tokens, per-token traffic can still beat dense. The reliable wins are per-token FLOPs and cache capacity; the reliable costs are memory floor and coordination." That's accurate and hedged.

Where MoE loses: memory floors, routing overhead, and small-batch regimes

"MoE is not free quality — it is a trade with three failure modes.

1. The memory floor. You must hold every expert in memory even though most are idle for any given token. DeepSeek-V3 in FP8 is ~671 GB of weights; Llama 4 Maverick is ~400 GB. If your fleet cannot hold the full model across accelerators, the per-token compute savings are unreachable.
2. Routing and communication overhead. Expert parallelism shards experts across GPUs, and every token's routing decisions generate all-to-all traffic between devices. At small batch sizes this overhead can dominate the compute you saved; MoE serving systems (custom kernels, expert-parallel collectives) are meaningfully more complex than dense serving.
3. Small-batch and fine-tune-heavy regimes. At batch size 1-4 the router's overhead, load imbalance, and the difficulty of keeping experts hot in cache erode the advantage. Teams doing frequent fine-tunes also face a less mature toolchain: balanced-routing losses, expert dropout, and evaluation quirks are active engineering, not solved problems.

Rule of thumb: MoE wins when concurrency is high, context is long, and you serve a shared base model. It loses when memory is scarce, batches are tiny, or your workflow is fine-tune-heavy."

Hmm, "load balancing/affinity" was mentioned in outline — I covered balancing. Fine.

Serving Software: The Multiplier You Control

"The same GPU can deliver 2-4× different throughput on the same model depending on the serving stack. Software is the multiplier you control, and it multiplies in both directions.

Continuous batching replaces static batches with a scheduler that admits new requests the moment any slot frees, so the GPU never waits for the slowest sequence in a batch. Prefix caching reuses the KV cache of shared prompt prefixes — system prompts, few-shot headers, RAG templates — so repeated context is computed once and paid for once; for agent fleets with a common system prompt, hit rates above 50% are common and prefill cost collapses accordingly. PagedAttention-style memory management (popularized by vLLM) eliminates the fragmentation that wastes 60-80% of KV cache in naive allocators. Scheduling policy — how you chunk prefill, prioritize interactive vs batch traffic, and place requests — decides whether TTFT stays under a second at load. And quantization interplay matters: FP8/INT4 shrink weights and cache, but each format changes which kernels and batching strategies are optimal.

-->

Software decides utilization. Two teams with identical GPUs, identical models, and identical traffic regularly report 2-4× throughput differences — the entire delta is batching, caching, and scheduling. Before you buy hardware to fix a capacity problem, benchmark your serving stack against your own traces."

Hmm, "hit rates above 50% are common" — plausible but I should hedge: "for agent fleets with a shared system prompt, prefix-cache hit rates can exceed half of all prefill tokens." Keep hedged with "can."

"fragmentation that wastes 60-80%" — the vLLM paper reported 60-80% memory waste in existing systems (KV cache waste due to fragmentation and reservation). Yes, the vLLM paper says existing systems waste 60-80% of KV cache memory. Good.

A Decision Framework for 2026

"Run this checklist before any capacity purchase:

1. Profile your context. What is the p50 and p99 context length, and what is the KV cache per token? Run the formula from the first section on your actual model config.
2. Measure concurrency, not volume. Requests per second is a marketing number; concurrent sequences × context length is what fills VRAM.
3. Decide your fine-tuning posture. Frequent fine-tunes or per-customer adapters push you toward dense or lightly-sparse models with mature tooling.
4. Check the memory ceiling. Total weights + peak concurrent KV cache + activations must fit with headroom; for MoE, that means the full expert set, sharded.
5. Benchmark with your traces. Replay real prompts against candidate stacks (vLLM, SGLang, TensorRT-LLM, your provider) and measure cost-per-token and TTFT at your p99, not a vendor's demo.

Adopt a portfolio: a sparse MoE workhorse for high-volume, long-context traffic; a dense or distilled model for latency-critical and fine-tuned paths; aggressive prefix caching everywhere.

Rule of thumb: run the KV cache math first, the MoE memory floor second, and the serving benchmark third. If a model fails the first two, no serving software saves it; if it passes, the serving stack sets your final cost-per-token.

The teams that win on cost-per-token in 2026 are not the ones with the newest GPUs. They are the ones that treated architecture as an economic decision from day one."

Hmm wait, the last line duplicates The Bottom Line. Move that sentiment to Bottom Line. End framework with the blockquote + portfolio line.

The Bottom Line

"Cost-per-token is set by three forces — KV cache memory, sparse activation, and serving software — and hardware is the third variable, not the first. Run the cache formula, respect the MoE memory floor, and tune the serving stack before you scale the fleet. Architecture is now an economic decision; make it deliberately."

Then italic validation note: "Figures are computed from public model cards and technical reports (Llama 3, DeepSeek-V2/V3, Qwen3, Llama 4) at standard precisions; recompute for your exact config, quantization, and kernel stack before committing budget."

Expert Q&A

4-5 Q&As for knowledgeable B2B readers:

Q1: "We're at 8K context today. Does the 128K math actually apply to us?" — A: cache scales linearly with context; at 8K the same 70B-class model holds ~2.6 GB per sequence (327,680 × 8192 ≈ 2.68 GB), so concurrency is cheap; but agent traces drift upward — model the p99, not the average, and note prefill O(n²) latency. Good.

Q2: "Does MLA make GQA obsolete?" — A: No. MLA is tied to DeepSeek's architecture; GQA remains standard in dense families; MLA requires specific kernel support (e.g., FlashAttention variants / custom decode kernels), and its benefit depends on recomputing K/V from latents — great at long context, adds a small compute tax at decode. Also portability: GQA models run on every mature stack.

Q3: "Our vendor quotes tokens/sec on a single stream. Why does our throughput collapse in production?" — A: single-stream numbers hide the memory-bound reality; throughput comes from batching; ask for tokens/sec at fixed concurrency and TTFT at p99 under load; single-stream numbers measure bandwidth, production cost is set by utilization.

Q4: "Is MoE worth it if we fine-tune monthly?" — A: hedged answer — LoRA on active paths, expert imbalance issues, tooling maturity; if fine-tune cadence is high and dataset small, dense 70B may serve better; consider distilling into dense for the fine-tuned path.

Q5: "Should we buy more GPUs or fix our serving stack first?" — A: benchmark first: 2-4× deltas from continuous batching + prefix caching are common; capacity math: KV cache formula, concurrency; only then buy. Also P/D disaggregation as mid-step.

Maybe also a question about FP8 vs INT4 quantization interplay? I have 5 solid ones. Let me finalize 5.

Q1: KV cache at shorter contexts / how to model. Q2: MLA vs GQA adoption. Q3: Vendor benchmark vs production throughput. Q4: MoE + fine-tuning. Q5: Buy hardware vs optimize stack (or quantization choice FP8 vs INT4 for long context). Actually a quantization question is very practitioner-relevant: "We quantize weights to INT4 to fit more users — why does long context still break?" A: weights are only half the story; KV cache dominates at long context; quantizing cache (KV cache quantization to FP8) helps but has quality risk; the formula includes bytes per parameter — halving weight bytes doesn't touch the 42 GB cache. That's a great Q&A. Let me use that as Q5 instead of buy-vs-optimize? I can do 5 Q&As total: pick the best 5.

Final set:

1. Context drift / p99 modeling (KV formula applied).
2. Quantization: INT4 weights but cache still kills us.
3. MLA vs GQA — should we wait for MLA models?
4. Vendor single-stream benchmarks vs production throughput.
5. MoE with frequent fine-tuning.

That's 5, all practitioner-real.

Word count check: Let me estimate the draft. Intro ~120, forces ~180, since-2024 ~200, attention intro ~35, KV math ~280, GQA/MLA ~300 + table, long context ~300, MoE intro ~0, routing ~230, tradeoff ~230 + table, where-MoE-loses ~250, serving ~280, framework ~250, bottom line ~70, note ~40. Total ≈ 2,545 + tables + Q&A ~380. Good — within target for the article body (~2,400-2,600).

Formatting details:

• Keep "Meta description:" and "Reading time:" lines at top (after title).
• Headings: # title, ## sections, ### subsections.
• --> blocks — three of them (attention variants, MoE vs dense, continuous batching). The outline explicitly planned these. Keep them as single lines in brackets.
• Bold: key SVO claims.
• Blockquotes: 42 GB callout, subscribe CTA, rule-of-thumb.

One more accuracy check on numbers I cite:

• "2 × 80 × 8 × 128 × 2 = 327,680 bytes ≈ 320 KB" — 327,680 bytes = 320 KiB. Saying "≈ 320 KB" fine.
• 327,680 × 131,072 = 42,949,672,960 bytes ≈ 42.9 GB (decimal GB). "~42 GB" or "~43 GB". The brief says ~42 GB. Use ~42 GB (and 40 GiB could be mentioned but skip).
• INT4 70B ≈ 35 GB; FP8 ≈ 70 GB. H100 80 GB. Correct.
• GQA 8 KV heads vs 64 query heads → 8× cut vs MHA. Correct.
• MQA 64× vs MHA. Correct.
• MLA ~93% reduction (93.3% per DeepSeek-V2). Correct.
• DeepSeek-V3: 671B/37B, 256 routed + 1 shared, top-8. Correct. Dec 2024. R1 Jan 2025. Correct.
• Qwen3-235B-A22B: 128 experts, 8 active. Correct.
• Llama 4 Maverick: 400B total, 17B active, 128 experts, 1 active + shared. I believe Maverick routing: each token → 1 shared + 1 routed? Hmm. Llama 4 Maverick: "128 experts... a token is routed to a single expert" — I'm fairly confident top-1 routed + shared expert. I'll phrase "activates one routed expert per token alongside a shared expert" — I'm fairly confident. To be safe I could omit Maverick's expert-activation detail and just give total/active params. Let me say "Llama 4 Maverick (~400B total, 17B active)" without expert counts. Safer.
• FlashAttention-3 ~2× over FA2 on Hopper FP16: FA3 paper reports 1.5-2× at FP16 (up to ~740 TFLOPS vs ~350-570 FA2). "up to ~2×" is fine.
• Gemma 3: 5:1