← research lab

Mac-local specialist field guide

How to evaluate a local LLM

Evaluate a local LLM with a pinned generation recipe, raw outputs and regression checks. Validate report files offline without loading a model.

A local LLM evaluation should answer a product decision, not merely produce a leaderboard number. Freeze the target and baseline first, run the baseline and candidate through the same protocol, inspect task-specific slices and regressions, then record a ship, route, retry, or reject decision.

Run a source-pinned generation smoke

This example uses PostTrainLLM source at 73da1aa and lm-eval 0.4.10. The commands are source-declared and have not been run together for this guide. They perform model inference; the offline report-card checker later on only checks evidence files.

Connect an existing local model

Use an installed serving binary and a model already on your machine. Starting the server loads that model. In one terminal, replace the placeholder with its actual path:

posttrainllm serve /absolute/path/to/existing-model \
  --host 127.0.0.1 --port 8765

In another terminal, check readiness. PostTrainLLM advertises the model ID posttrainllm; another compatible server may require a different ID. PostTrainLLM renders chat as ChatML-style text; confirm the model’s prompt format. A successful readiness response establishes the HTTP connection, not evaluation quality.

curl --fail --silent --show-error http://127.0.0.1:8765/v1/models

Pin the harness and choose a supported task

With Python 3.10 or newer, create a separate environment and install the API extras. This fixes the harness version used by the example. Record the model and dataset revisions separately; this is not a latest-version or safety recommendation.

python3 -m venv .venv-lm-eval
source .venv-lm-eval/bin/activate
python3 -m pip install "lm-eval[api]==0.4.10"

The pinned GSM8K task generates answers. Its chat adapter needs --apply_chat_template to send message objects. Use five examples, zero few-shot examples and a 128-token output cap for a plumbing check:

lm-eval --model local-chat-completions \
  --model_args "base_url=http://127.0.0.1:8765/v1/chat/completions,model=posttrainllm,tokenizer_backend=None,tokenized_requests=False,num_concurrent=1" \
  --tasks gsm8k --apply_chat_template \
  --num_fewshot 0 --limit 5 --batch_size 1 \
  --gen_kwargs "max_gen_toks=128,temperature=0.0" \
  --log_samples --output_path runs/quality-evals/gsm8k-local-smoke

Before running, check that each prompt plus the 128-token output allowance fits the model’s native context using its own tokenizer. PostTrainLLM’s --max-context only lowers the native limit. If any prompt is truncated, the run cannot support a scored comparison. Record stopped or incomplete answers and request failures. The harness may download task data. A metric from these five capped examples is a smoke result, not a full GSM8K benchmark or a baseline/candidate comparison.

Inspect raw results before reading the score

For this model ID, harness 0.4.10 writes results_<timestamp>.json and samples_gsm8k_<timestamp>.jsonl beneath runs/quality-evals/gsm8k-local-smoke/posttrainllm/. The result JSON records configuration and task metrics; the sample file preserves responses and filtered answers. Inspect both and keep the exact command, model/tokenizer identity, task revision and any failures.

These raw harness files use a different format from the site’s fine-tune report cards. Passing raw output to the report-card validator does not convert or qualify it. A report card needs its own provenance, paired gates, measurement states and decision.

Interpret failures and adapter limits

Open the local evaluation integration recipe for server options, wrapper behavior, output paths and the generation/log-probability boundary. Once the smoke path works in your authorized session, freeze the full target and regression evaluations described below.

Define the contract

Specify the inputs, expected outputs, allowed variation, protected content, and failure classes. Choose metrics that match the task: exact execution for SQL, tool and argument correctness for agents, unnecessary edit rate for autocorrection, or grounded answer criteria for retrieval. Keep a qualitative review lane, but do not let it overwrite deterministic gates.

Freeze the evaluation set

Separate training, validation, and test data. Record provenance and deduplicate across splits. A holdout that changes for every candidate cannot support a fair comparison. Mark synthetic fixtures as synthetic and avoid describing a small suite as production proof.

Run baseline and candidate symmetrically

Use the same prompts, decoding settings, runtime, and evaluator. Preserve raw predictions and measurement state. Record timeouts and missing values separately from zero. When the candidate is a routed specialist, evaluate both the target slice and the routing boundary.

Check paired prediction files before calculating a gain

For a small exact-string classification gate, keep one frozen gold row per task ID and one prediction row per ID for each model. Use status: "timeout" or "error" for attempted tasks that did not return an answer. A missing row is an integrity failure, not a smaller denominator. This simple format is separate from raw lm-eval output and report cards.

{"id":"case-1","expected":"file-listing"}
{"id":"case-2","expected":"abstain"}

Save those lines as gold.jsonl. Save the following as baseline.jsonl, and make a separate candidate.jsonl using the same IDs:

{"id":"case-1","status":"ok","output":"file-listing"}
{"id":"case-2","status":"timeout"}

Download compare-predictions.py and run it with Python 3. It uses only the standard library and loads no model:

python3 compare-predictions.py gold.jsonl baseline.jsonl candidate.jsonl

The baseline example scores 1/2 with one timeout. The checker rejects duplicate IDs, empty inputs, unknown statuses, missing successful outputs and differing ID sets. It compares strings exactly: whitespace and casing count. For SQL, tools or open-ended answers, replace the exact-match judge with a task-appropriate evaluator while retaining the ID and failure checks. A two-case synthetic fixture validates bookkeeping; it establishes no model-quality gain. The downloadable checker was exercised on valid and adversarial synthetic files for this guide.

Measure regressions explicitly

A specialist can improve the hard gate and still become worse outside it. Report deltas for every protected slice. Define reject conditions before the run, such as meaning changes, leakage, breadth loss beyond a threshold, or latency and memory outside the target machine budget.

Make the decision reproducible

Package the base and candidate identities, dataset revision, evaluator revision, configuration, per-slice results, missing measurements, and final decision. Keep “ship” narrower than “the candidate improved.” A routed ship can be correct when the target gain is real and the routing constraint is enforced.

PostTrainLLM includes fixture and no-GPU gates, live eval commands, canonical factory-run records, specialist packages, and public report cards. Several public results intentionally disclose missing historical latency, RAM, throughput, or raw traces rather than replacing those gaps with estimates.

Read a real local LLM evaluation report

Start with the retained Qwen3-4B ReST file-operations report card. It shows a target improvement alongside a broader regression. These are historical results, not a fresh evaluation performed for this guide. Open the report card and its machine-readable JSON.

Its file-operations gate rises from 0.75 to 1.00, while breadth falls from 0.667 to 0.556 (rounded to three decimal places). Both pairs are marked historical. A better score on the specialist task does not establish a better general assistant.

Check the base model, specialist artifact and recorded configuration first. Then inspect the target and regression gates, the evidence states, the decision and its verification blockers. Missing commands, frozen-eval identity or overlap checks cannot be inferred from a good score.

The card says routed-ship and verified: false: retain the file-operations specialist within its routing constraint, reject it as a general successor and do not use it as Pace’s default planner. A recorded decision and a schema-valid report do not establish a fully verified ship.

For your own evaluation, declare both a target gate and a protected regression gate before running the candidate. Preserve paired outputs and reject decisions when the candidate fails a requirement.

Validate the evidence format offline

From a clone of the PostTrainLLM repository, with Python 3 available, run the offline checker:

python3 scripts/factory/check_fine_tune_report_card.py browser/public/report-cards/qwen3-4b-rest-fused.json

This command validates the committed card’s schema, provenance states and decision consistency. It does not rerun a model evaluation or prove the historical scores. To reproduce a model result, you also need the exact dataset split, model and adapter revisions, evaluator, decoding settings and serving runtime. Name any missing input rather than estimating a new result.

Continue with the full local evaluation recipe or inspect the retained report cards before planning a paired comparison.

Tool-calling evaluation: preserve the denominator

Use the official BFCL definitions when reporting BFCL results. A selected local slice is not the full leaderboard. Publish the exact cases, scorer, model revision, decoding settings and failure policy; separate valid JSON from correct arguments, task completion and unintended actions.

Keep task depth and out-of-domain breadth separate. Our September ReST comparison reported 12/12 versus stock 9/12 on file operations, but 25/45 versus stock 30/45 on breadth. Neither denominator should disappear into a headline percentage. Repeated runs and uncertainty remain important even when a small gate scores perfectly.

After quantization, adapter fusion or export, repeat the same gate in the actual serving runtime. A smaller artifact does not prove lower runtime memory or higher throughput. Preserve rejected candidates in the attempt ledger so future readers can see what failed.