Distilling a Frontier Model Into 7B: What You Keep, What You Lose, What It Costs

The 7B Question Every AI Team Eventually Asks

A well-distilled 7B model can now match a 2023-era frontier model on a surprising range of production tasks — classification, extraction, formatting, tone-matched drafting — at a fraction of the per-token cost. That fact has launched a thousand "small models are all you need" posts. Most of them skip the fine print.

Model distillation is not a compression trick that preserves everything. It is an engineering budget decision: you trade some capability ceiling for a much lower marginal cost per request. The teams that get burned are the ones that assume the trade is free.

This article breaks the decision into three parts: what a 7B student reliably keeps from a frontier teacher, what degrades and how, and what the full cost stack looks like once you account for data generation, evaluation, and the maintenance you'll be doing a year from now.

[ILLUSTRATION: Side-by-side diagram of a large teacher model generating training data that flows into a small student model, with arrows labeled "logits," "sequences," and "eval gates"]

What Model Distillation Actually Transfers

Capability transfer vs. behavior transfer

In the teacher–student framing, a large model produces supervision and a smaller model learns to reproduce it. There are two main flavors, and the distinction matters operationally:

Logit-based (white-box) distillation trains the student on the teacher's full output distribution. The student minimizes a divergence — typically KL divergence — between its own softened logits and the teacher's, using a temperature parameter to smooth the distribution and expose the teacher's relative preferences among near-miss tokens. This carries far more signal per example than a single hard label, and it's the stronger method when you can use it. The catch: you need access to the teacher's logits, which generally means an open-weight or self-hosted teacher.
Sequence-level (black-box) distillation uses only sampled teacher outputs as targets. This is what you're stuck with when the teacher is a closed API — you get text, not distributions. It works, but it throws away the "dark knowledge" in the soft labels and typically needs more examples to reach comparable quality.

Neither method transfers the teacher's internal reasoning circuits. What moves across the gap is task behavior and style: how to format an answer, which tokens to prefer in context, what a good response looks like. The student learns to imitate the output surface, not the computation underneath it. It also inherits the teacher's errors — a student cannot exceed its teacher's ceiling on tasks the teacher gets wrong, so teacher-output quality is a hard constraint on student quality.

What you reliably keep

Some capabilities survive distillation remarkably well, especially when the task is bounded and the training data covers it:

Format adherence — JSON schemas, markdown structure, citation conventions
Instruction following on single-step, well-specified prompts
Domain classification and extraction — routing, tagging, entity pulls, field parsing
Tone and style matching — brand voice, register, response templates

On in-distribution, high-data-coverage tasks, narrow-task accuracy commonly lands within a few points of the teacher — often in the low single digits, though the exact gap depends on teacher quality, example volume, and how close your eval set sits to the distillation distribution. Treat any specific number as something you measure on your own eval, not something you assume. For a production classifier or extraction pipeline, that gap is usually acceptable. (See our guide to building task-specific eval sets before you commit to a distillation run.)

Where the 7B student holds its own

The 7B model wins on the economics of latency-sensitive, high-volume, well-scoped work. When you're processing millions of documents a month or serving sub-200ms responses, per-token cost and throughput dominate the decision. A distilled model you can host yourself also buys you data residency and predictable capacity — advantages a frontier API can't offer at any price.

The honest framing: a distilled 7B is not a smaller frontier model. It's a specialized instrument that happens to share a family resemblance.

What You Lose — The Honest Failure Modes

Long-horizon reasoning and multi-step planning

This is the first thing to break. Tasks requiring five or more dependent steps — multi-hop research, complex code refactors, agentic planning — show compounding error. As a simplifying illustration, if per-step accuracy were a stationary 95%, a ten-step chain would land near 60% end-to-end (0.95¹⁰ ≈ 0.60). Real chains are messier: errors correlate, and a single unrecovered mistake can poison every step after it.

The frontier teacher handles this better for a structural reason, not a magical one. It carries richer and more redundant intermediate representations, so it can detect that a step went wrong and self-correct before the error propagates. The student imitates the shape of reasoning without that redundancy — it has less capacity to notice its own mistakes, so it commits to them.

Test explicitly: build evals with 3-step, 5-step, and 8-step tasks and watch where the curve falls off a cliff. That cliff is your product boundary. (Our post on designing multi-step agent evals walks through the harness setup.)

Rare knowledge and tail distribution

Small models compress parametric knowledge aggressively. Long-tail facts — obscure regulations, niche APIs, regional specifics, recent events — are exactly what gets squeezed out first. Hallucination rates rise on the tail, and they rise quietly: the model sounds just as confident when it's wrong.

The fix is architectural, not a fine-tuning tweak. Pair the student with retrieval for anything fact-bearing, and treat its parametric knowledge as a fallback rather than a source of truth. (Related: choosing a retrieval stack for small-model deployments.)

Calibration and refusal behavior

Confidence scores drift after distillation. A student trained on teacher outputs inherits the teacher's expressed confidence, which may not match its own actual accuracy — the student can be confidently wrong in exactly the regions where it diverged from the teacher. Safety and refusal boundaries also shift: the student may over-refuse benign prompts or under-refuse edge cases the teacher handled correctly.

Measure this rather than assume it. Track expected calibration error (ECE) on a held-out set, abstention/refusal rates on a benign suite, and refusal-rate deltas on known edge cases. Re-tune guardrails against the student; don't assume the teacher's thresholds carry over.

Generalization beyond the distillation distribution

Off-distribution prompts expose the difference between imitation and understanding. Ask the student something structurally similar to its training data but semantically novel, and you'll see brittle pattern-matching where the teacher reasons. This is the failure mode that surprises teams in month three, when users start prompting in ways the eval set never anticipated. Budget for a held-out "novel phrasing" eval and re-run it every time you ship.

Distillation buys you a specialist. It does not buy you a generalist that happens to be small.

The Real Cost Stack

Upfront costs: data, compute, engineering

The sticker price is almost never the fine-tuning run. The real upfront costs break down roughly like this (order-of-magnitude, and heavily dependent on your teacher and data volume):

Teacher inference for synthetic data — generating 50k–500k high-quality examples from a frontier model is often the single largest line item. At typical frontier output pricing, hundreds of thousands of examples run into the hundreds or low thousands of dollars in teacher tokens alone — and you usually generate more than you keep after filtering.
Filtering and deduplication — raw teacher output is not training data. Expect to discard a meaningful fraction to dedup, quality filters, and format validation. This is engineering time, and it's the step teams most often underbudget.
Fine-tuning compute — a 7B LoRA or full fine-tune on a few hundred thousand examples is modest (single-digit to low-double-digit GPU-hours on modern accelerators). This is rarely the expensive part.
Evaluation and harness engineering — building task-specific evals, the multi-step harness, calibration checks, and a refusal suite. For most teams this exceeds the fine-tuning compute cost.

The recurring costs teams forget

The distillation run is a one-time event. The system is not:

Re-distillation on drift. When the teacher is upgraded, the task definition changes, or your input distribution shifts, you re-run the pipeline — and pay the data-generation cost again.
Eval maintenance. Evals rot. The novel-phrasing set especially needs refreshing as real usage drifts.
Guardrail re-tuning. Every re-distillation can shift calibration and refusal behavior, so guardrails get re-validated each cycle.
Serving infrastructure. Self-hosting buys residency and predictable capacity, but you now own the GPU capacity plan, autoscaling, and on-call.

One compliance flag

Before you train a student on a closed teacher's outputs, check the provider's terms of service. Using a frontier model's outputs to train a model that competes with it can violate usage terms, and the rules differ by provider. For a commercial product, get this reviewed — it's cheaper than discovering it after launch.

The Bottom Line

A distilled 7B is the right tool when your task is bounded, high-volume, latency-sensitive, and covered by your training data. It is the wrong tool when the task needs long-horizon reasoning, long-tail factual accuracy, or genuine generalization beyond what you trained on. Price the whole stack — data generation, filtering, evals, and the recurring maintenance — before you commit, and build the multi-step, tail, calibration, and novel-phrasing evals first. The evals are what tell you whether the trade is worth it, and they're the part you'll keep using long after the fine-tuning run is a memory.