A small language model specialist is a model that earns a narrow job through measured behavior. It does not need to outperform a frontier model everywhere. It needs to beat the chosen baseline on a defined target, stay inside regression and runtime limits, and route away when the task is outside its competence.
Pick the job
Good specialist targets have repeatable inputs, objective outcomes, enough representative data, and a runtime reason to stay local. Examples include tool selection, structured extraction, file operations, SQL generation, formatting, or a bounded intent router.
Establish the baseline
Run the unmodified base model on the frozen evaluation set before training. Record accuracy by slice, invalid outputs, latency, memory, and abstention or routing behavior when available. A specialist without a baseline has no demonstrated gain.
Choose the smallest intervention
Start with prompting or constrained decoding when it solves the problem. Use LoRA or QLoRA when behavior must be learned. Consider distillation when a stronger teacher can generate reviewed target behavior. Do not stack techniques until the failure requires them.
Protect breadth with routing
A specialist can regress on general tasks even while its target score improves. Add an explicit router or confidence boundary and evaluate it. The runtime should choose the specialist only for tasks it earned. Report a routed ship as routed, not as a new general default.
Write the route decision before reading the candidate score
For a proposed read-only file-listing specialist, the route could require an explicit list request and a path inside a user-approved directory. Deletion, ambiguous paths and general questions go to the unchanged fallback. That is an example product boundary, not a claim that the published specialist enforces it.
| Predeclared gate | Evidence required | Decision if missing or failed |
|---|---|---|
| Target behavior improves over unchanged base | Same frozen IDs, tool/argument judge and failure policy for both models | Retry data/training, or reject; lower loss is insufficient |
| No prohibited action in protected cases | Negative cases for writes, deletion, path escape and ambiguous intent | Reject route until the boundary is enforced and retested |
| Router selects only earned work | False-route and missed-route counts, including mixed and out-of-domain requests | Keep fallback as default; report router failure separately |
| Fallback remains usable | Actual serving-path test for abstention, timeout and specialist failure | Do not ship the route; a model score does not test fallback |
| Runtime stays within chosen device budget | Measured latency, peak memory and failure behavior for the serving artifact | Await evidence; do not fill with a parameter-count estimate |
Set task-specific thresholds and tolerated regression limits before the comparison. A passing target gate plus unknown routing or runtime evidence means “awaiting evidence,” not an unconditional ship. A narrow candidate may earn a route after all applicable gates pass, while still being rejected as a general replacement. Keep baseline-only, specialist-only and routed-system results separate so a good fallback cannot conceal a bad specialist.
Use the report-card contract to record the actual evidence and decision. This worksheet introduces no new experiment or result; it makes the proposed acceptance boundary reviewable before compute or data collection.
Package the whole result
A useful specialist package includes the base identity, adapter or fused artifact, tokenizer and prompt contract, routing rule, runtime requirements, license information, evaluation report, known regressions, and reproducible configuration. Missing latency or memory remains missing.
Make a decision
Use one of four honest outcomes: ship within the declared route, retry training, retry data or evaluation, or reject. Preserve failed attempts because they explain the next experiment and prevent the same recipe from being repeated without evidence.
PostTrainLLM is organized around this target-to-report loop. Its public file-operations specialist is a concrete example of a strong target gain paired with a disclosed breadth regression and a routing constraint.
CTA: Inspect the specialists and report cards, then follow the Mac fine-tuning workflow for your own target.
Route a specialist only where the evidence holds
A specialist package needs an explicit boundary: supported tasks, out-of-domain behavior, refusal conditions and the unchanged fallback. Teacher-generated trajectories are training material, not an independent test set. Deduplicate and split before training; freeze both a target gate and a breadth gate.
The published ReST package makes the tradeoff concrete. Its paired file-operations result improved from 9/12 to 12/12, while broader tool calling regressed from 30/45 to 25/45. The general-successor decision was reject. The tiny router also failed to generalize beyond its synthetic holdout.
These are inspectable learning artifacts, not a guarantee for your application. Start from the distillation recipe, preserve the evaluation provenance, and qualify routing and fallback on your own frozen workload before promotion.