Skip to content
PostTrainLLM docs
Esc
↑↓navigate↵open⌘Jpreview
On this page

LLM distillation: methods, commands, and fair comparisons

Compare logits distillation with response imitation, inspect local CLI prerequisites, and measure student quality, regressions, and costs fairly.

LLM distillation trains a student model using a teacher’s predictions or generated answers. It can help a smaller model learn a narrow task, but the gain depends on the teacher, data, student capacity, and evaluation. A lower training loss does not establish better task accuracy or cheaper inference.

This guide separates two methods, explains the local posttrainllm distill command, and gives a comparison checklist. For a recorded outcome, inspect the file-ops specialist report card. That is a response-imitation example with a breadth regression, not a measured comparison of the logits loss below.

Choose the method from the teacher access you have

Method What you need What the student learns from
Soft or logits distillation Teacher logits on the same token positions and vocabulary A probability distribution over next tokens
Response imitation Teacher-generated answers or checked tool trajectories The tokens in accepted responses, usually through supervised fine-tuning (SFT)
Gold-label SFT Verified target answers or executable reference trajectories Correct examples without needing a teacher model

Use logits distillation when a compatible teacher can run locally and its distribution is useful for the task. A text-only teacher API does not provide the full vocabulary distribution this implementation needs. Response imitation fits that access pattern: collect outputs, check them, then train in the student’s own tokenizer and chat format. It does not require a shared teacher tokenizer because teacher token probabilities are not being compared.

If you already have reliable gold answers, compare against gold-label SFT before paying to generate teacher responses. In every case, first check whether the unchanged student with a suitable prompt already meets the task.

Reasoning distillation is not automatically a logits loss

The DeepSeek-R1 paper, section 2.4, describes fine-tuning smaller models on curated responses using SFT. That is response imitation. Adding step-by-step text to a corpus does not by itself turn training into soft distillation, or demonstrate that a tiny student has acquired general reasoning ability. Test the required behavior on held-out tasks, including cases where an imitated answer would be wrong.

What the local logits command computes

For each next-token position, let p_t = softmax(teacher_logits / T) and p_s = softmax(student_logits / T). The executable loss in Distill.swift is:

L = alpha * T^2 * KL(p_t || p_s)
  + (1 - alpha) * NLL(student_logits, next_token)

KL(p_t || p_s) = sum_vocab p_t * (log(p_t) - log(p_s))

The direction is teacher to student. The teacher’s probabilities weight the discrepancy; the student is the trainable model. The hard-target NLL is ordinary next-token cross-entropy on the corpus. If the corpus is generated text, those targets are generated tokens, not independently verified truth.

Hinton, Vinyals, and Dean (2015), section 2, explains softened targets: increasing temperature spreads probability across alternatives, and the T^2 factor compensates for temperature’s gradient scaling. This motivates the loss; it does not predict a universal perplexity gain for an LLM or validate this repository’s implementation.

The current command defaults are --mode soft, --temperature 4, --alpha 0.7, --lr 3e-4, and --steps 1000. They are implementation defaults, not proven best settings. Alpha may range from 0 to 1; both terms are not mandatory. Temperature must be positive. Changing either is a recipe change whose task and regression effects need measurement.

Local command and prerequisites

The trainable student must be a from-scratch TinyGPTModel checkpoint loaded from .tinygpt; an HF-architecture student is rejected by this command. The teacher loader accepts a supported .tinygpt checkpoint or HF model directory. This is separate from the MLX LoRA response-imitation pipeline used by the file-ops specialist.

Before running soft distillation:

  1. Use an existing valid student checkpoint and compatible local teacher. Record their identities and the student’s prior training; do not hide an initialization run outside the student budget.
  2. Confirm the same tokenizer vocabulary and token IDs for both models. Equal vocabulary size is insufficient: token ID 42 must mean the same thing for each. Preserve tokenizer files and hashes. The runner checks vocabulary size only; it does not verify tokenizer identity or remap IDs.
  3. Supply a UTF-8 text corpus. --corpus is not a JSONL chat-data importer: JSONL passed here is tokenized as literal text. Keep evaluation examples outside this corpus and any teacher-data generation.
  4. Fit both models, logits, student gradients, and optimizer state within the memory budget. The teacher adds a forward pass per step. The overhead is model- and batch-dependent; there is no fixed 2× memory guarantee.

Illustrative command shape with placeholder paths, not a completed run:

posttrainllm distill ./student-init.tinygpt \
  --teacher ./teacher-model \
  --corpus ./train.txt \
  --tokenizer ./shared-tokenizer \
  --mode soft --temperature 4 --alpha 0.7 \
  --steps 1000 --batch 1 \
  --out ./student-distilled.tinygpt

For hard next-token training on already prepared teacher text, the command also supports the following shape. --mode hard defaults alpha to zero and does not load a teacher unless a positive alpha is explicitly supplied:

posttrainllm distill ./student-init.tinygpt \
  --corpus ./checked-teacher-text.txt \
  --tokenizer ./student-tokenizer \
  --mode hard --alpha 0 --steps 1000 --batch 1 \
  --out ./student-imitation.tinygpt

This hard mode uses NLL over the entire text. It does not add assistant-only masking, generate responses, or verify their correctness. For chat/tool trajectories, inspect the specialist distillation recipe and its retained data-format and evaluation evidence instead. Saving a checkpoint is not a quality or runtime qualification.

Curate response-imitation data before training

A teacher response is a proposal. Keep the original record and its review outcome so rejected generations do not disappear from the experiment history. For a synthetic formatting task, one candidate record could look like this:

{"id":"teacher-1","prompt":"Return the label for a request to list files.","response":"file-listing","source":"synthetic-fixture-v1","teacher_revision":"fixture-no-model-called","license_reviewed":true,"private_data_reviewed":true,"verified":true,"split":"train"}

The revision is explicitly a fixture identity, not a real teacher model. For actual generations, record the teacher/model revision, generation configuration, source identity and reviewer or executable check that justified acceptance. verified: true records that decision; setting a boolean does not perform a review. Keep evaluator evidence separately. Do not let a teacher or reviewer see the sealed test set while making training examples.

Download curate-teacher-records.py, save candidate records as teacher-records.jsonl, then run with Python 3:

python3 curate-teacher-records.py teacher-records.jsonl > review-decisions.jsonl

This dependency-free offline example emits one acceptance decision per ID. It does not call a teacher, load a student, write training examples or train a model. It fails on malformed JSON, empty input and duplicate/missing IDs. It rejects records with absent prompt/response/provenance, missing affirmative license, privacy or verification review, rows outside the training split, and exact prompt/response duplicates of an already accepted example.

Candidate Review output What to do next
Complete, reviewed record above accepted: true Inspect the associated check evidence before export
Same answer with verified: false needs_verified Run the task-specific judge or review; do not turn the flag on to pass
A second ID with the same accepted prompt and response duplicate_accepted_example Keep the rejection and deduplicate before sampling
A row marked split: "test" outside_training_split Keep the holdout outside teacher-data generation and training

The example was exercised on valid and adversarial synthetic fixtures, including false/string review flags, missing provenance, duplicate content, duplicate IDs and holdout rows. It checks explicit review bookkeeping and exact duplicates; it cannot detect private data, establish licensing, judge answer quality or find paraphrases. Group-related sources and task templates before splitting, using the trajectory dataset preflight when episodes are the unit.

Only after acceptance should you convert the response into the student’s training format. For the pinned MLX LM chat format, map prompt and response to user and assistant messages and apply the student’s chat template; provenance stays in a sidecar manifest. Prompt masking is a separate training choice in that source version, not something this filter adds. Measure the unchanged student and the adapted student on the untouched target and regression gates. An accepted corpus is input evidence, not a model quality result.

A comparison you can interpret

Start with the factory evaluation protocol. Write the question before choosing the budget: are you testing whether teacher information helps the same student, or whether a smaller student can replace a larger teacher within a resource limit?

  • Freeze data and scoring. Record train/validation/test identities, overlap checks, task scorer, prompts, decoding settings, and regression slices before tuning. Teacher generation must not see the sealed test set. Benchmark grades here require a qualified frontier ceiling; an unqualified set may still expose relative regressions but cannot establish parity.
  • Control initialization. For a logits-versus-NLL ablation, branch both candidates from the exact same student checkpoint. Hold training corpus, tokenizer, batch, context, optimizer, and sampling policy fixed where possible; record seeds and differences. Keep the unchanged student and teacher as evaluation anchors.
  • Name the budget. Equal student steps can control token exposure when batches and context match. They are not matched compute: soft distillation also runs a teacher forward pass. For a fixed-time or fixed-compute comparison, measure that budget and record the different steps/tokens completed. Do not describe one protocol as the other.
  • Separate costs. Record teacher pretraining or acquisition separately from teacher response/logit generation, student adaptation, and evaluation. State which costs are shared, sunk, unavailable, or included. Measure student serving on the same hardware, runtime, precision, prompt lengths, and output limits; parameter count alone does not establish latency or RAM.
  • Decide from behavior. Inspect target scores, error cases, safety and breadth. Lower perplexity on the same held-out corpus/tokenizer may support a language-modeling result; it does not substitute for tool-use or reasoning accuracy. Route a narrow winner only when the router and regression policy support that decision; otherwise retry or reject.

Keep this small result worksheet with the artifact. Use not measured for missing evidence rather than an expected number:

Record Required comparison
Model and recipe identity Baseline/student/teacher revisions; tokenizer; seed; changed variables
Quality and regressions Same frozen task and slices; denominators; scores; failures; decision
Training cost Student steps/tokens/time/peak RAM; teacher work and data cost separately
Serving cost Paired latency, peak RAM, tok/s, hardware/runtime/precision; or not measured
Scope What the evidence supports, missing checks, and stop/routing conditions

Retained example: a file-ops gain with a breadth cost

The file-ops report card and retained evaluation report preserve historical results from June 2026, not a fresh evaluation. They record a Qwen3-4B specialist adapted from the same 4B base using accepted trajectories. The 12-task hard gate is recorded as 58% to 100%. The 52-task out-of-domain breadth comparison falls from 59.6% to 42.3%; its frontier ceiling was not validated, so this is a relative regression signal, not a frontier-capability grade. The retained verdict is routed file-ops use only, not general planning.

Read the dated method and failure analysis for trajectory filtering, chat-template formatting, and stop-signal handling. This response-imitation result does not measure soft KL against NLL, student-versus-scratch perplexity, smaller-student compression, or matched compute. Paired serving RAM, latency, and tok/s are not recorded in that evaluation report. No numerical gain for this guide’s logits command is established by that artifact.

Next: inspect the specialist recipe, then read the report card and its caveats. Use the checklist above to identify what an existing result does and does not demonstrate before starting a new experiment.

Was this page helpful?