← research lab

Mac-local specialist field guide

How to build a small language model specialist

Build a routed small language model specialist with a frozen baseline, narrow training objective, regression gates, packaging, and an honest decision.

A small language model specialist is a model that earns a narrow job through measured behavior. It does not need to outperform a frontier model everywhere. It needs to beat the chosen baseline on a defined target, stay inside regression and runtime limits, and route away when the task is outside its competence.

Pick the job

Good specialist targets have repeatable inputs, objective outcomes, enough representative data, and a runtime reason to stay local. Examples include tool selection, structured extraction, file operations, SQL generation, formatting, or a bounded intent router.

Establish the baseline

Run the unmodified base model on the frozen evaluation set before training. Record accuracy by slice, invalid outputs, latency, memory, and abstention or routing behavior when available. A specialist without a baseline has no demonstrated gain.

Choose the smallest intervention

Start with prompting or constrained decoding when it solves the problem. Use LoRA or QLoRA when behavior must be learned. Consider distillation when a stronger teacher can generate reviewed target behavior. Do not stack techniques until the failure requires them.

Protect breadth with routing

A specialist can regress on general tasks even while its target score improves. Add an explicit router or confidence boundary and evaluate it. The runtime should choose the specialist only for tasks it earned. Report a routed ship as routed, not as a new general default.

Write the route decision before reading the candidate score

For a proposed read-only file-listing specialist, the route could require an explicit list request and a path inside a user-approved directory. Deletion, ambiguous paths and general questions go to the unchanged fallback. That is an example product boundary, not a claim that the published specialist enforces it.

Predeclared gateEvidence requiredDecision if missing or failed
Target behavior improves over unchanged baseSame frozen IDs, tool/argument judge and failure policy for both modelsRetry data/training, or reject; lower loss is insufficient
No prohibited action in protected casesNegative cases for writes, deletion, path escape and ambiguous intentReject route until the boundary is enforced and retested
Router selects only earned workFalse-route and missed-route counts, including mixed and out-of-domain requestsKeep fallback as default; report router failure separately
Fallback remains usableActual serving-path test for abstention, timeout and specialist failureDo not ship the route; a model score does not test fallback
Runtime stays within chosen device budgetMeasured latency, peak memory and failure behavior for the serving artifactAwait evidence; do not fill with a parameter-count estimate

Set task-specific thresholds and tolerated regression limits before the comparison. A passing target gate plus unknown routing or runtime evidence means “awaiting evidence,” not an unconditional ship. A narrow candidate may earn a route after all applicable gates pass, while still being rejected as a general replacement. Keep baseline-only, specialist-only and routed-system results separate so a good fallback cannot conceal a bad specialist.

Use the report-card contract to record the actual evidence and decision. This worksheet introduces no new experiment or result; it makes the proposed acceptance boundary reviewable before compute or data collection.

Package the whole result

A useful specialist package includes the base identity, adapter or fused artifact, tokenizer and prompt contract, routing rule, runtime requirements, license information, evaluation report, known regressions, and reproducible configuration. Missing latency or memory remains missing.

Make a decision

Use one of four honest outcomes: ship within the declared route, retry training, retry data or evaluation, or reject. Preserve failed attempts because they explain the next experiment and prevent the same recipe from being repeated without evidence.

PostTrainLLM is organized around this target-to-report loop. Its public file-operations specialist is a concrete example of a strong target gain paired with a disclosed breadth regression and a routing constraint.

CTA: Inspect the specialists and report cards, then follow the Mac fine-tuning workflow for your own target.

Route a specialist only where the evidence holds

A specialist package needs an explicit boundary: supported tasks, out-of-domain behavior, refusal conditions and the unchanged fallback. Teacher-generated trajectories are training material, not an independent test set. Deduplicate and split before training; freeze both a target gate and a breadth gate.

The published ReST package makes the tradeoff concrete. Its paired file-operations result improved from 9/12 to 12/12, while broader tool calling regressed from 30/45 to 25/45. The general-successor decision was reject. The tiny router also failed to generalize beyond its synthetic holdout.

These are inspectable learning artifacts, not a guarantee for your application. Start from the distillation recipe, preserve the evaluation provenance, and qualify routing and fallback on your own frozen workload before promotion.