Distilling a Frontier LLM into a Small Model: A Teacher-Student Playbook for Production

You shipped an LLM feature. It works. Then the bill arrives: a ticket-triage classifier, an intent router, or a field extractor is calling a frontier model on every request, adding a second of latency and a line item finance keeps asking about. The task is narrow. The model is general. You are paying for capability you never use.

Distillation fixes this. You use the large model as a teacher to label your real traffic, then train a small student model on those labels. The student handles the common path cheaply; the teacher stays as a fallback for hard cases.

This post is a practical playbook: when distillation is worth it, how to build the pipeline, how to evaluate the student honestly, and where teams get burned.

When distillation is the right move

Distillation pays off when the task is narrow, high-volume, and stable.

Signal	Distill?	Why
Classification, routing, tagging, extraction with a fixed schema	Yes	Small models match large ones on bounded tasks
100k+ calls/month on the same prompt	Yes	Fixed training cost amortises quickly
Latency budget under ~300 ms	Yes	Small models run locally or on cheap GPUs
Open-ended generation, multi-step reasoning	Usually no	Students lose the long tail of capability
Task definition changes weekly	No	You'll retrain constantly
Fewer than ~1k examples of real traffic	Not yet	Collect data first

Also check your provider's terms: some restrict using model outputs to train competing models. Read them before you build the pipeline, and prefer open-weight teachers if in doubt.

The pipeline at a glance

Collect real, unlabelled inputs (production logs, with PII redacted).
Label them with the teacher, using your best prompt.
Filter noisy labels (low confidence, disagreement across samples).
Gold set: hand-label a few hundred examples. Never train on this.
Train the student on teacher labels.
Evaluate the student against the gold set, not against the teacher.
Route: student first, teacher fallback on low confidence.
Monitor drift and re-label periodically.

The most common mistake is skipping step 4 and measuring the student by agreement with the teacher. That tells you how well you copied the teacher's mistakes.

Step 1-3: Label with the teacher, then filter

Use structured output and sample the teacher more than once. Disagreement across samples is a cheap, useful noise signal.

import json
from collections import Counter
from concurrent.futures import ThreadPoolExecutor

LABELS = ["billing", "bug", "feature_request", "account", "other"]

SYSTEM = f"""Classify the support ticket into exactly one label from {LABELS}.
Respond with JSON: {{"label": "<label>"}}"""

def teacher_label(client, text: str, n: int = 3) -> dict:
    votes = []
    for _ in range(n):
        resp = client.chat(  # replace with your provider's SDK call
            model="teacher-model",
            temperature=0.7,
            system=SYSTEM,
            user=text,
        )
        try:
            votes.append(json.loads(resp)["label"])
        except (json.JSONDecodeError, KeyError):
            continue
    if not votes:
        return {"label": None, "agreement": 0.0}
    label, count = Counter(votes).most_common(1)[0]
    return {"label": label if label in LABELS else None,
            "agreement": count / n}

def label_dataset(client, texts, workers=8):
    with ThreadPoolExecutor(workers) as pool:
        return list(pool.map(lambda t: teacher_label(client, t), texts))

Then keep only examples where the teacher agreed with itself:

rows = [
    {"text": t, **lab}
    for t, lab in zip(texts, labels)
    if lab["label"] and lab["agreement"] >= 0.67
]
print(f"kept {len(rows)}/{len(texts)}")

Dropped examples are not wasted: they are your hard set. Review a sample by hand. They often reveal ambiguous label definitions, which is a product problem, not a model problem.

Step 4: Build a gold set you never train on

Hand-label 300-500 examples, stratified by class, sampled from real traffic. Two people labelling a subset lets you measure inter-annotator agreement. If humans only agree 80% of the time, no model will be reliably better than that, and you should tighten the label guide before training anything.

Keep the gold set in a separate file and guard it:

import hashlib

def is_gold(text: str, gold_hashes: set[str]) -> bool:
    return hashlib.sha256(text.encode()).hexdigest() in gold_hashes

train_rows = [r for r in rows if not is_gold(r["text"], gold_hashes)]

Deduplicate near-duplicates between train and gold too, otherwise leakage will flatter your numbers.

Step 5: Train the student

For classification, a small encoder (a DistilBERT or MiniLM-class model) fine-tuned with Hugging Face transformers is usually the best cost/quality point. For structured extraction, a 1-3B parameter decoder with LoRA is more typical (see our LoRA/QLoRA guide).

from datasets import Dataset
from transformers import (AutoTokenizer, AutoModelForSequenceClassification,
                          TrainingArguments, Trainer)

model_name = "distilbert-base-uncased"
tok = AutoTokenizer.from_pretrained(model_name)
label2id = {l: i for i, l in enumerate(LABELS)}

ds = Dataset.from_list([
    {"text": r["text"], "label": label2id[r["label"]]} for r in train_rows
]).train_test_split(test_size=0.1, seed=42)

def encode(batch):
    return tok(batch["text"], truncation=True, max_length=256)

ds = ds.map(encode, batched=True)

model = AutoModelForSequenceClassification.from_pretrained(
    model_name, num_labels=len(LABELS),
    id2label={i: l for l, i in label2id.items()}, label2id=label2id,
)

args = TrainingArguments(
    output_dir="student",
    learning_rate=3e-5,
    per_device_train_batch_size=32,
    num_train_epochs=3,
    eval_strategy="epoch",
    save_strategy="epoch",
    load_best_model_at_end=True,
)

Trainer(model=model, args=args, tokenizer=tok,
        train_dataset=ds["train"], eval_dataset=ds["test"]).train()

Argument names change between transformers versions (for example evaluation_strategy became eval_strategy), so check the docs for the version you have installed.

A refinement worth knowing: instead of hard labels, train on the teacher's probability distribution (soft labels) with a KL-divergence loss. This transfers more information per example, but only works when you can get calibrated probabilities from the teacher, which API models often don't expose. Majority-vote hard labels are the pragmatic default.

Step 6: Evaluate against gold, by slice

from sklearn.metrics import classification_report
import numpy as np

preds = trainer.predict(gold_ds).predictions.argmax(-1)
print(classification_report(gold_labels, preds, target_names=LABELS))

Compare three numbers on the same gold set: teacher, student, and a trivial baseline (majority class or keyword rules). Then look at slices: per class, per customer segment, per input length. Students typically fail on rare classes and long inputs first.

A reasonable ship criterion: student within 1-3 points of the teacher on overall F1, and no class dropping more than 5 points.

Step 7: Route with confidence, keep the teacher as fallback

You rarely need to replace the teacher entirely. A cascade captures most of the savings with little quality loss.

import torch

def classify(text: str, threshold: float = 0.85):
    inputs = tok(text, return_tensors="pt", truncation=True, max_length=256)
    with torch.no_grad():
        probs = model(**inputs).logits.softmax(-1)[0]
    conf, idx = probs.max(0)
    if conf >= threshold:
        return {"label": LABELS[idx], "source": "student", "conf": conf.item()}
    return {**teacher_label(client, text, n=1), "source": "teacher"}

Tune the threshold on the gold set: plot accuracy against the fraction of traffic the student handles, and pick the knee. Raw softmax confidence is often over-confident, so calibrate it (temperature scaling is a simple fix) before trusting a threshold.

Log every fallback. Those rows are your next training batch, labelled for free.

Cost and latency: the comparison to run

Do the arithmetic before you start. Fill in your own numbers.

	Teacher (API)	Student (self-hosted)
Cost per 1M requests	tokens x price	GPU/CPU hours
p50 latency	hundreds of ms to seconds	typically tens of ms
Ops burden	none	hosting, monitoring, retraining
Quality ceiling	high	bounded by task and data
Fixed cost	none	labelling + training + eval engineering

If your monthly teacher spend is small, the engineering time may not pay back. Distillation is an optimisation, not a starting point.

Common pitfalls

Measuring against the teacher. Agreement is not accuracy. Always use human gold labels.
Gold-set leakage. Duplicates and near-duplicates across splits inflate scores silently.
Training on unfiltered labels. Noisy teacher outputs cap student quality. Filter by self-agreement.
Class imbalance. The student learns the majority class well and ignores the tail. Rebalance or weight the loss.
Distribution shift. Production inputs change (new product, new customers). Schedule re-labelling of a recent sample and watch the fallback rate; a rising rate is your drift alarm.
Ignoring the label guide. If the categories are fuzzy, both teacher and student will be inconsistent. Fix definitions first.
Privacy. Redact PII before sending logs to a teacher or storing training data.
Over-distilling. Not every task compresses. If the student plateaus far below the teacher, the task needs the larger model; keep the cascade.

A checklist before you ship

Task is narrow, stable, and high-volume
Provider terms allow training on outputs
300+ hand-labelled gold examples, held out and deduplicated
Teacher labels filtered by self-agreement
Student compared to teacher and baseline, sliced by class
Confidence calibrated; threshold chosen from the gold set
Teacher fallback in place, with logging
Drift monitor and a retraining cadence

Next steps

Start small: pick your highest-volume, simplest LLM call, export a week of redacted inputs, and run the labelling script above. Spend an afternoon on the gold set; it will pay for itself in every later decision. Even if you never ship the student, the exercise tells you exactly how much of your traffic really needs a frontier model.

If you want to build this end to end with feedback from practitioners, our AI Engineering cohort walks through evals, fine-tuning, and production routing on real projects.