Speculative Decoding in Production: What the Speedups Actually Look Like

The Gap Between Speculative Decoding Benchmarks and Production Reality

Speculative decoding has moved from a research curiosity to a standard feature in production LLM serving stacks. vLLM, TensorRT-LLM, and TGI all ship it as a configuration option, and vendor benchmarks routinely advertise 1.5x to 3x decoding speedups. For engineering leaders planning inference capacity, that range is tantalizing.

But the number in a launch blog post and the number you observe in your own environment are rarely the same. Realized speculative decoding speedups depend on the model pair, the workload, your batch sizes, and how carefully you tune acceptance behavior. Teams that deploy it expecting a uniform 2x win often find something closer to 1.3x–1.5x on latency-sensitive single-stream traffic—and, on their busiest, most heavily batched endpoints, sometimes discover that throughput quietly regressed. Speculative decoding is not free: the draft model consumes GPU compute and KV-cache memory that would otherwise serve more concurrent requests.

This article separates the marketing from the mechanics. We'll cover why speculative decoding matters for latency, how it works inside a serving stack (including the math that predicts its speedup), what realistic production numbers look like, how acceptance rate and workload shape the outcome, and exactly what to measure before and after rollout. The goal is a grounded picture you can use to set expectations and design a rollout that survives contact with real traffic.

Why Speculative Decoding Matters for Inference Latency

Autoregressive decoding is memory-bandwidth-bound, not compute-bound. Generating a single token requires a full forward pass through the model, and at typical batch sizes the GPU spends most of that pass waiting on weight and KV-cache memory reads rather than doing useful arithmetic. The hardware is underutilized, but the serial dependency between tokens means you can't simply parallelize your way out of it.

That serial dependency is what users feel. Inference latency—measured as time-to-first-token (TTFT) and inter-token latency (ITL)—directly shapes perceived responsiveness. A chatbot that streams tokens slowly feels broken even when aggregate throughput is healthy. And because GPU time is expensive, latency and cost per request are tightly coupled: the longer each request occupies the accelerator, the more you pay to serve it.

Speculative decoding is one lever among several. Continuous batching improves throughput (see our guide to [LLM inference optimization techniques](internal: /blog/llm-inference-optimization)), quantization reduces memory pressure, and KV-cache optimization (paged attention, prefix caching) cuts redundant work. Speculative decoding targets a different bottleneck: the number of sequential forward passes required to emit a sequence of tokens. By letting a smaller model guess ahead and a larger model verify multiple tokens at once, it attacks the serial dependency directly.

Two clarifications matter for planning:

• It is primarily a latency optimization, not a throughput optimization. It reduces the number of sequential target-model passes for a given output length. Whether that converts into higher throughput depends entirely on whether your GPU had spare compute headroom for the draft model to use.
• It does not help TTFT and can slightly hurt it. The first token still requires one full prefill and one target-model step; the draft model adds a small amount of per-request overhead. The metric that moves is ITL (and therefore total generation time for long outputs). If your workload is dominated by short outputs or prefill-heavy prompts, speculative decoding is the wrong lever.

How Speculative Decoding Works in an LLM Serving Stack

The core idea is straightforward. A small draft model proposes several tokens ahead. The target model—the large model you actually care about—then verifies all those proposals in a single forward pass, accepting the tokens that match what it would have generated and rejecting the rest. Because the target model processes the draft tokens in parallel, you can emit multiple accepted tokens per target-model pass.

Critically, with the correct rejection-sampling scheme, the output distribution is preserved. The tokens you emit are statistically indistinguishable from what the target model would have produced on its own. Speculative decoding is quality-neutral when implemented as exact rejection sampling—it is a pure speed technique, not a quality tradeoff.

One caveat that trips up teams: greedy/argmax speculative decoding is not distribution-preserving. Many frameworks default to a greedy verification path for speed. It is a different decoding algorithm than exact rejection sampling and can shift outputs, especially on open-ended generation. If output equivalence matters (regulated domains, regression-tested prompts), confirm your stack is using sampling-based verification with a fixed seed, or validate the greedy path against your eval suite before rollout.

Draft Models, Target Models, and Token Verification

The draft and target models are typically drawn from the same family—a 1B–8B draft paired with a 70B target, for example—so their tokenizers and output distributions align. Vocabulary and tokenizer identity is the hard constraint: mismatched tokenizers force vocabulary remapping and collapse acceptance rate. The draft model runs several cheap forward passes to produce a candidate sequence; the target model then runs one forward pass over the whole candidate sequence and checks each token against its own distribution.

The economics are simple: you win when the draft model is much cheaper than the target model and its proposals are usually right. Draft size is a tuning knob. A larger draft model proposes better tokens but costs more per proposal and eats more memory; a smaller one is cheap but noisier. Verification cost scales with candidate length, so the number of speculative tokens is also a tradeoff, not a free multiplier.

The draft-model approach is not the only option, and the alternatives matter for stack selection:

• EAGLE / EAGLE-2 / EAGLE-3 draft at the feature (hidden-state) level rather than the token level, achieving higher acceptance rates than a standalone small model at lower draft cost.
• Medusa adds multiple decoding heads to the target model itself, avoiding a separate draft model.
• N-gram / lookahead decoding uses no draft model at all—it proposes continuations from the prompt and recent context. This is extremely cheap and performs well on input-grounded tasks (summarization, RAG, code editing) where output tokens are copied from the input, but does little for open-ended generation.
• Self-speculative / layer-skip methods use a subset of the target model's own layers as the drafter.

For a deeper walkthrough of the verification algorithm, see our explainer on [how speculative decoding works](internal: /blog/how-speculative-decoding-works).

The Math That Predicts Your Speedup

Acceptance rate is the single most important variable governing realized speedup—but it interacts with draft length (γ) and the cost ratio between draft and target. The governing relationship is worth internalizing.

With per-token acceptance probability α and draft length γ, the expected number of tokens accepted per target-model pass is:

E[tokens per step] = (1 − α^(γ+1)) / (1 − α)

The wall-clock speedup is then approximately:

Speedup ≈ E[tokens per step] / (1 + γ · c_draft)

where c_draft is the cost of one draft forward pass as a fraction of one target forward pass. (Verification of γ tokens in one target pass is roughly the cost of a single target step, which is the whole point.)

Two consequences fall out immediately:

• Diminishing returns on γ. At α = 0.8, going from γ = 3 to γ = 5 raises expected tokens per step only modestly while increasing draft cost linearly. The optimum γ is usually 3–5, not 8–10.
• Low α is catastrophic. At α = 0.5 with γ = 4, expected tokens per step is ~1.9, but you paid 4 draft passes to get there. If c_draft is meaningful, you can end up slower than plain decoding. This is the mechanism behind the "latency improved but throughput regressed" reports.

Acceptance rate is workload-dependent, which is why the same configuration can shine on one endpoint and disappoint on another. A draft model paired with a target model on creative writing may accept 60% of tokens; on structured JSON extraction or code completion it may accept 85%+.

"Speculative decoding doesn't make your model faster in the abstract—it makes your model faster on the traffic where a small model can predict a large model's next tokens."

What Realistic Production Speedups Look Like

Setting expectations requires separating the regimes. The numbers below are representative of well-tuned deployments on modern datacenter GPUs (H100/A100-class) and should be treated as planning ranges, not guarantees—your acceptance rate and batch profile will move them.

Regime	Typical ITL speedup	Notes
Single-stream, low concurrency, high acceptance (α ≈ 0.8+)	1.8x–2.5x	The regime vendor benchmarks report. Memory-bandwidth-bound.
Single-stream, moderate acceptance (α ≈ 0.6–0.7)	1.3x–1.7x	Common for open-ended chat with a generic draft.
Batch size near compute saturation	~1.0x or negative	Target pass is compute-bound; draft model steals FLOPs and KV memory.
Prefill-heavy / short-output workloads	~1.0x	TTFT unchanged; too few decode steps to amortize.
Input-grounded tasks with n-gram/lookahead	2x–4x on affected spans	Cheap drafter, very high acceptance on copied text.

The pattern to internalize: speculative decoding pays off precisely where you are memory-bandwidth-bound and acceptance is high, and it costs you where you are compute-bound. The same cluster can show a 2x win on a low-traffic interactive endpoint and a throughput regression on the high-QPS batch endpoint that actually drives your bill.

How Acceptance Rate and Workload Shape the Outcome

Acceptance rate is a property of the pair of models and the distribution of traffic, not a fixed constant. Three factors dominate:

1. Model alignment. Same-family draft and target, matched tokenizers, and a draft trained or distilled on similar data maximize α. A generic small model paired with an unrelated large model will underperform.
2. Task structure. High-entropy generation (creative writing, reasoning) is hard for a small model to predict; low-entropy, high-structure output (JSON, code, templated responses, extractive summaries) is easy. If your traffic is mixed, acceptance rate will vary by request—measure the distribution, not just the mean.
3. Temperature and sampling. Higher sampling temperature lowers expected acceptance because the target's distribution is flatter. If you serve high-temperature traffic, expect lower α and tune γ down accordingly.

The practical implication: do not enable speculative decoding globally and assume uniform benefit. Segment your traffic, measure acceptance rate per segment, and enable it where the math works. For many teams the right answer is to enable it on interactive, latency-sensitive endpoints and leave high-throughput batch endpoints on plain decoding.

What to Measure Before and After Rollout

Speculative decoding is easy to enable and easy to mis-evaluate. A credible rollout measures the following, holding concurrency and traffic mix constant between the control and treatment arms:

Before rollout (baseline):

• ITL p50/p99 and TTFT p50/p99, per endpoint and per traffic segment.
• Throughput (tokens/sec and requests/sec) at your real production concurrency, not a benchmark batch size.
• GPU utilization and memory headroom, to know whether you are memory- or compute-bound.

After rollout (treatment):

• Acceptance rate — and its distribution across requests, not just the mean. A mean of 0.75 hiding a bimodal 0.9/0.4 split means your tuning is wrong for half your traffic.
• Accepted tokens per target pass — the direct driver of latency; compare against the formula above to see whether γ is set well.
• ITL p50/p99 — the metric speculative decoding is supposed to move. If ITL didn't improve, the configuration is not paying for itself.
• Throughput at production concurrency — the metric most likely to regress. Watch it explicitly.
• Draft-model resource tax — additional KV-cache memory and FLOPs consumed, which reduces the batch size you can sustain.

A/B methodology: run control and treatment against a shadow copy of production traffic (or a fixed replay set) with sampling seeds pinned, so output differences are attributable to the greedy-vs-exact verification path rather than to sampling noise. Hold concurrency fixed; if you let batching drift, you cannot separate latency effects from throughput effects.

The teams that get the most from speculative decoding are not the ones with the largest advertised multiplier—they are the ones who know their acceptance-rate distribution, set γ accordingly, and enable the feature only on the endpoints where the memory-bandwidth math favors it.