"We Cut Inference Spend 60% Without Changing Models": An Infrastructure Lead on What Actually Worked

The Setup: A 60% Inference Cost Reduction With the Same Models

When Dana Whitfield took over infrastructure at a mid-size B2B SaaS company in early 2025, the inference bill was the fastest-growing line item in her budget—and nobody could tell her exactly where it was going. By the end of that year, her team had cut monthly inference spend by roughly 60% against a 90-day baseline. They did it without swapping a single model, without fine-tuning, and without a quality regression beyond their agreed tolerance on their internal eval suite.

That last part is what makes the story worth telling. The default advice for reducing AI inference costs is to shop for a cheaper model or shrink prompts until something breaks. Whitfield's team did neither. They treated inference spend as an infrastructure and observability problem, and most of the wins came from plumbing that never touched the model weights.

"Everyone wants the fix to be a smarter model or a cleverer prompt. Ours was a cache, a queue, per-request cost visibility, and a router. That's it. That's the whole trick." — Dana Whitfield, infrastructure lead

The company asked us to refer to it generically and to withhold a few vendor names. The numbers and the sequence of decisions are as she described them; we have not independently audited the spend figures or the eval results. Below is what worked, what flopped, what it cost to get there, and what she'd tell a team starting this quarter.

[ILLUSTRATION: A simple line chart showing monthly inference spend trending down ~60% over six months while request volume climbs steadily upward, with annotations marking when observability, caching, batching, and routing were each introduced.]

What Their Inference Stack Looked Like Before

Scale and spend profile

At the start of the cost project, the company was running about 4.5 million inference requests per day across a mix of interactive product features and background jobs. By token volume, roughly 70% went to a mid-tier hosted model, 20% to a smaller fast model, and 10% to a frontier model reserved for the hardest tasks. Monthly inference spend sat at just under $180,000, spread across two providers. That's not hyperscaler money, but it was growing 15–20% per month over the prior two quarters, and the finance team wanted a plan before it doubled again.

Why the inference bill crept up

The growth wasn't purely demand. Three things were quietly inflating spend:

Retries without backoff discipline. Failed calls were retried at the application layer with no deduplication, so transient provider errors occasionally produced three or four billed calls for one user action.
Duplicate prompts. Support summarization, classification, and a dozen internal tools were re-sending near-identical requests thousands of times a day with no shared cache.
Over-provisioning. Every feature defaulted to the largest model that "worked," and nobody had revisited those choices since launch.

"None of these were dramatic," Whitfield said. "Each one was maybe 10–15% of waste. But they stacked, and they compounded as volume grew."

The fix that came first: per-request cost visibility

Before touching a single lever, the team instrumented spend. Every request got tagged with a feature ID, a task class, a model, and a token count, and those tags rolled up into a per-feature cost dashboard. This is the unglamorous step most teams skip, and it's the one that made everything after it measurable.

"You cannot cut what you can't attribute," Whitfield said. "We thought support summarization was our biggest cost center. The dashboard said it was bulk enrichment running on the frontier model because of a config default from eighteen months earlier. We'd have spent a month optimizing the wrong thing."

The cost model was reconciled against provider invoices monthly so the dashboard tracked actual billed spend, not an estimate.

The 60% Breakdown: Where the Inference Savings Actually Came From

No single lever delivered the 60%. The savings compounded, and the order mattered—each fix made the next one easier to measure.

The attribution below is a de-duplicated contribution to the total delta, not a set of independent percentages that happen to add up. Because the levers overlap (a cached call is also a call you don't route or batch), the team's finance partner reconciled them sequentially against the 90-day baseline, holding request volume constant and removing each lever's effect before measuring the next. The shares are approximate and will not transfer cleanly to a different workload mix.

Lever	Share of total savings (approx., de-duplicated)	What it addressed
Exact + semantic caching	~25 points	Duplicate and near-duplicate prompts
Routing, right-sizing, waste removal	~20 points	Wrong model, unbounded retries, over-provisioning
Batching and request coalescing	~15 points	Per-call overhead on background jobs

The three levers above account for the bulk of the cut; the remainder came from the retry and config fixes folded into the routing line and from smaller cleanups. If you do the naive multiplicative math on 25%, 15%, and 20%, you land near 49%, not 60%—the gap is why the reconciliation was done sequentially rather than by multiplying independent reductions, and why we're presenting shares of a measured total rather than standalone percentages.

Caching (exact + semantic)

Caching was the single largest contributor. Exact-match caching handled the boring majority: repeated system prompts, idempotent classification calls, and identical support queries, keyed on a hash of the normalized prompt plus model and parameters.

Semantic caching—matching requests by embedding similarity above a tuned threshold—caught the paraphrased duplicates that exact matching missed. This is the lever with the most quality risk, and the team treated it accordingly:

Threshold tuning was empirical, not guessed. They set the similarity threshold by sweeping it against a labeled set of known-duplicate and known-distinct request pairs, then chose the point that maximized true cache hits while keeping false-positive hits at effectively zero. The threshold was re-checked whenever the embedding model or prompt templates changed.
A strict exclusion list. Anything personalized, tenant-specific, time-sensitive, or with side effects was never semantically cached—only exact-cached where safe. No cache entry was shared across tenants.
TTLs and invalidation. Cache entries carried short TTLs and were invalidated on prompt-template version bumps, so a stale answer couldn't outlive the logic that produced it.
Cache hits were eval-gated like any other change. A false-positive semantic hit is a wrong answer served instantly, so hit quality was sampled and scored continuously, not just at launch.

Together the two cache tiers removed roughly a quarter of total spend, and the hit rate kept climbing as the team widened coverage—but only within those guardrails. (See our guide to LLM caching strategies for production workloads.)

Routing, right-sizing, and killing waste

The team built a lightweight router that sent each request to the smallest model that reliably passed evals for that task class, escalating only when a confidence signal was low. They also fixed retry logic (bounded retries with exponential backoff and request-level deduplication, so a transient error no longer produced three or four billed calls) and reset feature defaults that had drifted to the frontier model. This category was less glamorous but delivered about 20 points—much of it from simply not paying for the frontier model on tasks a smaller one handled fine.

"The router was the part I was most skeptical of, because it's where quality regressions hide. We gated every routing change behind the eval suite. If pass rate dipped beyond our tolerance, the change didn't ship." — Whitfield

The confidence signal matters here. A router that escalates on a poorly calibrated score will either over-escalate (and save nothing) or under-escalate (and ship bad answers). The team calibrated the escalation threshold on held-out traffic before enabling it, and kept a kill switch to force a task class back to its previous model. (More on this in our primer on LLM routing and model right-sizing.)

Batching and request coalescing

Background workloads—nightly summarization, bulk enrichment, embedding generation—were firing one request at a time and paying full per-call overhead. Moving them to async batched queues and coalescing near-simultaneous identical requests cut that segment's cost by a wide margin. Two guardrails made this safe:

Coalescing only on a normalized, non-personal key. Requests were merged only when they were byte-identical after normalization and carried no tenant-specific or user-specific data. Nothing was ever coalesced across tenants, and nothing with a side effect was merged.
Non-interactive only. No user ever waited on a batch; interactive traffic kept its own path.

How They Measured Quality

"No measurable quality drop" is easy to say and hard to defend, so here is the bound the team actually used. They maintained a fixed internal eval suite per task class—a few hundred labeled cases for the high-traffic classes, fewer for the long tail—and treated a change as shippable only if pass rate stayed within a pre-agreed tolerance of the frozen baseline. Human spot-checks ran weekly on a sample of production outputs, including cached and routed responses, specifically to catch the failure modes the eval suite might miss.

The honest caveat: once routing decisions are gated on an eval suite, that suite becomes a target, and a static suite can drift from real traffic. The team mitigated this by refreshing eval cases from production samples quarterly and by tracking a small set of business metrics (support-deflection rate, classification accuracy on a live sample) alongside the eval score. Quality was held, not proven invariant—and the bound was defined before the project started, not after.

What It Cost to Save

A 60% cut didn't come free, and the bill has two parts.

New infrastructure. The team added a cache store, a vector index for semantic matching, and a queue for batched jobs. These carry their own monthly cost—well under the savings, but not zero—and they added operational surface area: another set of services to monitor, version, and page on. The net saving is the 60% gross figure minus this added infra spend.

Engineering effort. The bulk of the work was roughly two engineer-quarters spread across the observability build, the cache tiers, the queue migration, and the router—with the router and semantic cache taking the most time because both required eval work, not just code. The observability dashboard came first and paid for itself by preventing a month of optimizing the wrong feature.

For a team deciding whether to attempt this: the levers are real, but the sequencing (visibility → safe caching → batching → routing) is what kept the effort from being wasted. Skipping straight to a router without per-request cost attribution is how you optimize the wrong thing confidently.

What Didn't Work (And What They'd Skip Next Time)

Whitfield is candid that roughly a third of the team's early effort produced nothing shippable.

Aggressive quantization. They experimented with heavily quantized self-hosted variants to chase lower per-token cost. Quality drift showed up on the eval suite immediately, and the ops burden of maintaining the serving stack ate the projected savings.
Premature self-hosting. They stood up a self-hosted inference cluster before utilization justified it. At their volume the fixed cost of the cluster—GPU hours, idle capacity, and the on-call rotation to keep it healthy—outran the per-token savings, and they wound it down. Self-hosting pays off at sustained high utilization; below that, it's a cost center with a pager.
Chasing the last few points of cache hit rate. Widening semantic-cache coverage past a point started surfacing false-positive risk faster than it added savings, so they stopped. The marginal hit wasn't worth the quality exposure.

The pattern: the wins came from not paying for things twice and not paying for more model than the task needed. The losses came from trying to win on unit economics (cheaper per-token compute) before the workload was efficient enough to deserve it.

What She'd Tell a Team Starting This Quarter

Instrument before you optimize. Per-request cost attribution by feature and task class, reconciled to invoices. Do this first or you'll optimize the wrong line item.
Fix waste before you fix unit cost. Retries, config defaults, and over-provisioning are free money and carry no quality risk.
Cache in two tiers, with guardrails. Exact first, semantic second—and only with an exclusion list, TTLs, and hit-quality sampling. Never semantically cache personalized or side-effecting calls.
Batch only what no human waits on, and coalesce only on non-personal keys.
Gate every routing change behind evals you defined before you started, and keep a kill switch.
Budget the savings net of the new infra and the engineering time. The gross number is the headline; the net number is the business case.

"The 60% wasn't a model decision. It was a discipline decision—see the spend, stop paying twice, and stop using a bigger model than the task needs. The models never changed. Our understanding of them did." — Dana Whitfield