- Home
- Skills
- Education & Learning
- Knowledge Distillation: Teacher to Student Models
More screenshots
Works with the AI Agents you already use
Knowledge Distillation: Teacher to Student Models
Compress a large model into a smaller, cheaper student that keeps what matters: soft-target training, the right KL direction, staged size ratios, and an eval loop that proves it worked.
$5
Knowledge Distillation: Teacher to Student Models
Example session with this skill installed
We serve a large in-house model as a code-review assistant and the inference bill is out of hand. We want a student roughly 8x smaller for first-pass reviews: it has to keep catching the real bugs, keep its explanations readable, and know when to escalate to the big model instead of guessing. We can run the teacher locally. Plan the distillation.
- Read your context and instructions
- Compiled the knowledge distillation
Inputs check: teacher runs locally with logits available, so this is Recipe 1 (logit distillation) — not response distillation, which would cap quality for no infrastructure reason. Student: an instruction-tuned model ~8x smaller, inside the 7–15x sweet spot, so a single-stage run is approved; no intermediate staging needed.
Capability list, frozen before training: (1) true-positive rate on seeded real bugs, (2) false-alarm rate (a noisy student teaches devs to ignore it — ranked equal to #1), (3) escalation calibration: saying "needs the senior model" on the cases it would get wrong, (4) explanation readability, scored on a fixed sample.
KL direction: forward KL. First-pass review is classification-like — the student should concentrate on the teacher's most likely verdicts per hunk, not cover its full generative diversity. Generation-style diversity is deliberately not a ranked capability here.
Data plan: 60k real review cases from history. 70% teacher-labeled (soft targets cached in one no-grad pass, reused across epochs), 30% ground truth from merged/rejected outcomes, with escalation cases oversampled — they are rare and they are capability #3. Alpha 0.7, temperature 2, 2 epochs, LR 2e-5.
Pilot first: 4k cases, full eval harness (which already exists, because it was built before training started). Watch the soft loss separately: if it stalls while hard loss falls, the student is memorizing outcomes instead of absorbing judgment, and the pilot dies here for the price of an afternoon.
Acceptance: fidelity 90%+ agreement with the teacher at temperature 0 on held-out cases; bug catch rate within 3 points of the teacher; false-alarm rate no worse than 2 points above the teacher; escalation precision audited by hand on 200 cases. Failure audit then clusters the divergences — expectation: long cross-file reasoning chains — and those clusters become the escalation policy, not a surprise in production. Ship gate at 8+/10 on the readiness rubric, with the serving win verified on the real deployment target.
Connects securely to your tools. The creator never sees your data.
About this skill
Distillation trains a small model on a big model's full output distribution — including its second and third choices — instead of one-hot labels, so the student absorbs the teacher's judgment rather than its answers. Done by habit instead of by design, it produces a student that matches the teacher on training data and disappoints in production. This skill turns it into an engineering run. It opens with an inputs/outputs contract and a stop rule (no capable teacher, or a student already inside budget, means there is nothing to distill). The core mechanism is covered precisely: temperature scaling, the T²-corrected soft loss, the hard-loss blend, and the choice that matters most — forward KL for classification-style tasks, reverse KL so open-ended generation does not collapse to one safe answer. Four recipes are given with a selection table: logit distillation when you own the teacher, response distillation for API-only teachers, multi-teacher when no single teacher covers your capability list, and two-stage when a sharp task skill must follow general transfer. A six-step run protocol freezes the capability list and eval set before training, baselines the student, and forces a cheap pilot run before the full one. The hyperparameters that actually move outcomes (data mix, temperature, the 7–15× size-ratio sweet spot, alpha against a teacher's known bad habits) are set by table, and the proof section defines fidelity measurement at two temperatures, benchmark deltas on ranked capabilities, and a failure audit whose clusters become the next data recipe. A shipping rubric (8+/10) and a complete worked example — a support-answer model taken 10:1 smaller with refusals intact — close it out.
Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for Education & Learning.
- It plans and evaluates distillation; it does not run training jobs or ship training code for your stack.
- It is not a quantization or pruning method — those compress without a teacher-student training run and need different guidance.
- Results depend on teacher quality and corpus coverage; no recipe transfers a capability the teacher does not actually have.
- Teacher outputs and distilled students can carry licensing or terms-of-use constraints — the skill flags the check, but the terms review is yours.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before buying a skill.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Money-back guarantee
Doesn't do what its page promises? You get your money back.
Trust & safety
Security scan
Security scan passed 24 days ago
Permissions
No special permissions declared or detected