Skip to content
PostTrainLLM docs
Esc
↑↓navigate↵open⌘Jpreview
On this page

lm-evaluation-harness integration for PostTrainLLM

Source-pinned lm-eval 0.4.10 local generation recipe, adapter limits, and an offline inspector for retained results and raw samples.

Use a compatible local server with a task its adapter can score. This reference checks source at PostTrainLLM 73da1aa1be9293cdbd75b825e0793fe786c65a5f and EleutherAI lm-evaluation-harness 0.4.10, commit f7d0b116146bd616b59d0b991549120f197369a4. The commands below are source-declared examples, not a newly executed model evaluation. The version pin is for reproducibility, not a latest-version or safety recommendation.

For the shorter task guide and retained report-card interpretation, start at Evaluate a local LLM. A fresh model run in this repository still follows the fresh-experiment rule.

Match task, endpoint and tokenizer

Generation, such as gsm8k (generate_until)

local-chat-completions, /v1/chat/completions

Pass --apply_chat_template; confirm the model’s prompt format and context budget.

Likelihood scoring, such as HellaSwag or ARC

A suitable completion or native scoring backend

The chat adapter raises NotImplementedError for loglikelihood. A completion backend needs matching tokenization and prompt log probabilities.

PostTrainLLM’s source exposes /v1/models, /v1/chat/completions and /v1/completions. Its plain completion handler has a separate logprobs plus echo scoring path. That source surface does not establish end-to-end HellaSwag or ARC correctness for a chosen tokenizer, model or harness release. Changing the URL alone does not qualify a likelihood evaluation.

The chat handler renders messages as ChatML-style text. Check that format against the model’s training contract. For another prompt format, choose a compatible text-completion or native backend and validate the formatting.

Sources: the pinned chat and completion adapters, API interface guide, and PostTrainLLM server.

Prepare an existing local server

Use an already built posttrainllm binary and a model already on disk. CONTRIBUTING.md owns build prerequisites. The main CLI already dispatches serve; no dispatch edit is required. Starting it loads the selected model:

posttrainllm serve /absolute/path/to/existing-model \
  --host 127.0.0.1 --port 8765

--lora /path/to/adapter.lora selects an adapter on the existing base. --max-context N only caps context below the model’s native limit; it cannot increase the trained context capacity. Record every option used, the model and adapter identities, tokenizer, binary revision and machine. Check readiness from another terminal:

curl --fail --silent --show-error http://127.0.0.1:8765/v1/models

The PostTrainLLM model ID is posttrainllm. Use the advertised model ID when pointing at another compatible server. Readiness establishes an HTTP response, not correct generation or a benchmark result.

Pin the harness and run a bounded generation smoke

Harness 0.4.10 requires Python 3.10 or newer. Its api extra supplies the HTTP dependencies; see the pinned package metadata.

python3 -m venv .venv-lm-eval
source .venv-lm-eval/bin/activate
python3 -m pip install "lm-eval[api]==0.4.10"

The pinned GSM8K definition uses generate_until and normally requests five few-shot examples. This smoke explicitly uses zero, limits the task to five examples and caps each response at 128 generated tokens:

lm-eval --model local-chat-completions \
  --model_args "base_url=http://127.0.0.1:8765/v1/chat/completions,model=posttrainllm,tokenizer_backend=None,tokenized_requests=False,num_concurrent=1" \
  --tasks gsm8k --apply_chat_template \
  --num_fewshot 0 --limit 5 --batch_size 1 \
  --gen_kwargs "max_gen_toks=128,temperature=0.0" \
  --log_samples --output_path runs/quality-evals/gsm8k-local-smoke

Check prompt lengths with the model’s own tokenizer before running. Every prompt plus the output allowance must fit the native context. If any prompt is truncated, the run cannot support a scored comparison. The harness may download task data, and the command performs real inference. Five capped examples check protocol and artifact flow; they do not establish a full GSM8K score, baseline/candidate gain or performance budget. Preserve stop conditions, incomplete answers, timeouts and missing measurements.

Optional repository wrapper

python_ref/lm_eval_tinygpt.py can spawn the server, wait for /v1/models, run the harness and stop its own server on exit. With an existing server, use an explicit generation task and pass the chat/sample flags through:

python3 python_ref/lm_eval_tinygpt.py --skip-spawn \
  --base-url http://127.0.0.1:8765/v1/chat/completions \
  --tasks gsm8k --num-fewshot 0 --limit 5 --batch-size 1 \
  --output-path runs/quality-evals/gsm8k-local-smoke \
  --lm-eval-extra "--apply_chat_template --log_samples --gen_kwargs max_gen_toks=128,temperature=0.0"

The wrapper always selects local-chat-completions. Its default tasks, and the defaults in evals/run_quality_evals.sh, are HellaSwag and ARC-Easy. Those defaults request unsupported likelihood scoring through this chat adapter. Do not use them as a working recipe; choose the explicit generation path above. The wrapper accepts a model path, --posttrainllm-bin, --port and --max-context when spawning. Its compatibility smoke-target fallback is not evidence about the current installed binary.

Keep and inspect raw artifacts

For model ID posttrainllm, harness 0.4.10 writes beneath runs/quality-evals/gsm8k-local-smoke/posttrainllm/:

  • results_<timestamp>.json contains aggregate metrics and configuration.
  • samples_gsm8k_<timestamp>.jsonl contains raw responses and filtered answers when --log_samples is set.

See the pinned output writer. The wrapper’s final summary search checks only the top output directory. Its no results_*.json found message can therefore miss the model subdirectory; inspect the files and the harness exit status before concluding that no results were written.

Raw harness JSON and JSONL are not the site’s fine-tune report-card schema. The offline check_fine_tune_report_card.py validator does not convert these files, run a model or prove a score. A canonical factory report needs its own provenance, paired target/regression gates, measurement states and decision; see the report-card contract.

Inspect a retained harness run without inference

When a score file exists, first confirm that the corresponding raw samples are present. Download inspect-harness-artifacts.py and run it with Python 3 against one retained run directory:

python3 inspect-harness-artifacts.py \
  runs/quality-evals/gsm8k-local-smoke/posttrainllm gsm8k 5

The final argument is the expected number of evaluated examples from the run’s command, not a number inferred from whichever rows survived. Use 5 only for the five-example smoke above. For a full or differently limited task, supply its declared expected count and preserve the command that establishes it.

The inspector uses the pinned harness’s results task mapping and logged doc_id, resps and filtered_resps fields. These are raw harness artifacts, not the fine-tune report-card schema. It requires exactly one result JSON and one task sample JSONL under the selected directory; archives with several runs must be separated rather than silently choosing the newest. It checks that the task has metric fields, sample IDs are unique integers, raw and filtered responses exist, and the sample count matches the declared count.

For example, a synthetic schema fixture, not a measured result, can have this result file named results_fixture.json:

{"results":{"gsm8k":{"exact_match,none":0.0}},"config":{"model":"fixture-no-inference"}}

And one line in samples_gsm8k_fixture.jsonl:

{"doc_id":0,"resps":[["fixture answer"]],"filtered_resps":["fixture answer"]}

Run the inspector with expected count 1 for that fixture. It prints the task, count, metric keys and the two artifact paths. The zero in the fixture is only placeholder data; the inspector does not recompute or endorse that metric. The script was exercised on this schema shape and duplicate IDs, absent raw responses, count mismatches, absent task metrics and ambiguous multi-run folders, without loading a model or downloading a dataset.

A structural pass is only a preflight. Manually inspect the saved configuration, resolved task/model revisions, prompt and chat template, stop/truncation behavior, per-sample outputs and scoring filters. The checker cannot establish that two files came from the same process, that the expected IDs are the correct dataset split, that an answer is correct, or that the recorded metric matches a recomputation. Preserve those provenance and evaluator checks before claiming a benchmark or comparison. A five-sample smoke remains a smoke even when every artifact check passes.

Source contract: the harness sample writer and sample-record construction, release 0.4.10. Recheck this inspector’s assumptions before using another release. The paired prediction example is a separate exact-string format; raw harness files cannot be substituted for it.

Interpret failures before interpreting scores

  • Readiness or connection failure: check the process, bind host, port and URL.
  • A message-list assertion: include --apply_chat_template and use the full chat-completions URL.
  • loglikelihood not implemented: the task and adapter do not match. Choose a suitable scoring backend with matching tokenizer and log-probability support.
  • Truncated prompts or incomplete generations: preserve the failure and the context/output limits. Do not compare the result with an uncapped full run.
  • Missing artifacts or a nonzero harness exit: inspect server/harness logs. Missing scores remain missing, not zero.

Freeze the full target and regression suite, dataset revision, prompts, decoding settings and acceptance thresholds before a paired comparison. Record latency, RAM and throughput only when measured under that protocol. No wall-clock estimate or expected model score is qualified by this recipe.

Retained HTTP smoke evidence

The historical documentation retained these responses from the separate posttrainllm-serve-smoke executable and a 27M checkpoint. They show the recorded HTTP response shapes and stop behavior. They do not qualify the current main binary, model quality or a completed lm-eval run.

$ curl -s http://127.0.0.1:8765/v1/models
{"object":"list","data":[{"object":"model","id":"posttrainllm","owned_by":"posttrainllm"}]}

$ curl -s -X POST http://127.0.0.1:8765/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{"model":"posttrainllm","messages":[{"role":"user","content":"Once upon a time"}],"max_tokens":10,"temperature":0.0}'
{"id":"chatcmpl-...","object":"chat.completion",
 "choices":[{"message":{"role":"assistant","content":"The first step in the process is to make a"},
             "finish_reason":"stop","index":0}],
 "model":"posttrainllm","created":...,
 "usage":{"prompt_tokens":13,"completion_tokens":10,"total_tokens":23}}

$ curl -s -X POST http://127.0.0.1:8765/v1/completions \
    -H "Content-Type: application/json" \
    -d '{"prompt":"Once upon a time","max_tokens":15,"temperature":0.0}'
{"object":"text_completion",
 "choices":[{"text":" of time, the time of time is not always a matter of time.",
             "finish_reason":"stop","index":0}],
 "usage":{"prompt_tokens":4,"completion_tokens":15,"total_tokens":19}}

# Stop sequence smoke — generation cuts off before "time" appears.
$ curl -s -X POST http://127.0.0.1:8765/v1/completions \
    -H "Content-Type: application/json" \
    -d '{"prompt":"A","max_tokens":50,"stop":["time"],"temperature":0.0}'
{"object":"text_completion",
 "choices":[{"text":". 1998.\n- \"The first ", "finish_reason":"stop", …}], …}

Source files

  • native-mac/Sources/TinyGPT/TinyGPT.swift: main serve dispatch.
  • native-mac/Sources/TinyGPTServe/Serve.swift: HTTP handlers, chat formatting and completion scoring.
  • python_ref/lm_eval_tinygpt.py: spawn/reuse and argument forwarding.
  • evals/run_quality_evals.sh: retained driver with incompatible chat-task defaults described above.

For the benchmark landscape and contamination discussion, use the retained benchmark research. Historical research claims do not substitute for qualification on a chosen runtime.

Was this page helpful?