lm-evaluation-harness integration for PostTrainLLM
Source-pinned lm-eval 0.4.10 local generation recipe, adapter limits, and an offline inspector for retained results and raw samples.
Use a compatible local server with a task its adapter can score. This reference
checks source at PostTrainLLM 73da1aa1be9293cdbd75b825e0793fe786c65a5f
and EleutherAI lm-evaluation-harness 0.4.10, commit
f7d0b116146bd616b59d0b991549120f197369a4. The commands below are
source-declared examples, not a newly executed model evaluation. The version
pin is for reproducibility, not a latest-version or safety recommendation.
For the shorter task guide and retained report-card interpretation, start at Evaluate a local LLM. A fresh model run in this repository still follows the fresh-experiment rule.
Match task, endpoint and tokenizer
Generation, such as gsm8k (generate_until)
local-chat-completions, /v1/chat/completions
Pass --apply_chat_template; confirm the model’s prompt format and context budget.
Likelihood scoring, such as HellaSwag or ARC
A suitable completion or native scoring backend
The chat adapter raises NotImplementedError for loglikelihood. A completion backend needs matching tokenization and prompt log probabilities.
PostTrainLLM’s source exposes /v1/models, /v1/chat/completions and
/v1/completions. Its plain completion handler has a separate logprobs plus
echo scoring path. That source surface does not establish end-to-end
HellaSwag or ARC correctness for a chosen tokenizer, model or harness release.
Changing the URL alone does not qualify a likelihood evaluation.
The chat handler renders messages as ChatML-style text. Check that format against the model’s training contract. For another prompt format, choose a compatible text-completion or native backend and validate the formatting.
Sources: the pinned chat and completion adapters, API interface guide, and PostTrainLLM server.
Prepare an existing local server
Use an already built posttrainllm binary and a model already on disk.
CONTRIBUTING.md
owns build prerequisites. The main CLI already dispatches serve; no dispatch
edit is required. Starting it loads the selected model:
posttrainllm serve /absolute/path/to/existing-model \
--host 127.0.0.1 --port 8765
--lora /path/to/adapter.lora selects an adapter on the existing base.
--max-context N only caps context below the model’s native limit; it cannot
increase the trained context capacity. Record every option used, the model
and adapter identities, tokenizer, binary revision and machine. Check readiness
from another terminal:
curl --fail --silent --show-error http://127.0.0.1:8765/v1/models
The PostTrainLLM model ID is posttrainllm. Use the advertised model ID when
pointing at another compatible server. Readiness establishes an HTTP response,
not correct generation or a benchmark result.
Pin the harness and run a bounded generation smoke
Harness 0.4.10 requires Python 3.10 or newer. Its api extra supplies the
HTTP dependencies; see the pinned package metadata.
python3 -m venv .venv-lm-eval
source .venv-lm-eval/bin/activate
python3 -m pip install "lm-eval[api]==0.4.10"
The pinned GSM8K definition
uses generate_until and normally requests five few-shot examples. This smoke
explicitly uses zero, limits the task to five examples and caps each response
at 128 generated tokens:
lm-eval --model local-chat-completions \
--model_args "base_url=http://127.0.0.1:8765/v1/chat/completions,model=posttrainllm,tokenizer_backend=None,tokenized_requests=False,num_concurrent=1" \
--tasks gsm8k --apply_chat_template \
--num_fewshot 0 --limit 5 --batch_size 1 \
--gen_kwargs "max_gen_toks=128,temperature=0.0" \
--log_samples --output_path runs/quality-evals/gsm8k-local-smoke
Check prompt lengths with the model’s own tokenizer before running. Every prompt plus the output allowance must fit the native context. If any prompt is truncated, the run cannot support a scored comparison. The harness may download task data, and the command performs real inference. Five capped examples check protocol and artifact flow; they do not establish a full GSM8K score, baseline/candidate gain or performance budget. Preserve stop conditions, incomplete answers, timeouts and missing measurements.
Optional repository wrapper
python_ref/lm_eval_tinygpt.py can spawn the server, wait for /v1/models,
run the harness and stop its own server on exit. With an existing server, use
an explicit generation task and pass the chat/sample flags through:
python3 python_ref/lm_eval_tinygpt.py --skip-spawn \
--base-url http://127.0.0.1:8765/v1/chat/completions \
--tasks gsm8k --num-fewshot 0 --limit 5 --batch-size 1 \
--output-path runs/quality-evals/gsm8k-local-smoke \
--lm-eval-extra "--apply_chat_template --log_samples --gen_kwargs max_gen_toks=128,temperature=0.0"
The wrapper always selects local-chat-completions. Its default tasks, and
the defaults in evals/run_quality_evals.sh, are HellaSwag and ARC-Easy.
Those defaults request unsupported likelihood scoring through this chat
adapter. Do not use them as a working recipe; choose the explicit generation
path above. The wrapper accepts a model path, --posttrainllm-bin, --port
and --max-context when spawning. Its compatibility smoke-target fallback
is not evidence about the current installed binary.
Keep and inspect raw artifacts
For model ID posttrainllm, harness 0.4.10 writes beneath
runs/quality-evals/gsm8k-local-smoke/posttrainllm/:
results_<timestamp>.jsoncontains aggregate metrics and configuration.samples_gsm8k_<timestamp>.jsonlcontains raw responses and filtered answers when--log_samplesis set.
See the pinned output writer.
The wrapper’s final summary search checks only the top output directory.
Its no results_*.json found message can therefore miss the model subdirectory;
inspect the files and the harness exit status before concluding that no
results were written.
Raw harness JSON and JSONL are not the site’s fine-tune report-card schema.
The offline check_fine_tune_report_card.py validator does not convert these
files, run a model or prove a score. A canonical factory report needs its own
provenance, paired target/regression gates, measurement states and decision;
see the report-card contract.
Inspect a retained harness run without inference
When a score file exists, first confirm that the corresponding raw samples are present. Download inspect-harness-artifacts.py and run it with Python 3 against one retained run directory:
python3 inspect-harness-artifacts.py \
runs/quality-evals/gsm8k-local-smoke/posttrainllm gsm8k 5
The final argument is the expected number of evaluated examples from the run’s
command, not a number inferred from whichever rows survived. Use 5 only for
the five-example smoke above. For a full or differently limited task, supply
its declared expected count and preserve the command that establishes it.
The inspector uses the pinned harness’s results task mapping and logged
doc_id, resps and filtered_resps fields. These are raw harness artifacts,
not the fine-tune report-card schema. It requires exactly one result JSON and
one task sample JSONL under the selected directory; archives with several runs
must be separated rather than silently choosing the newest. It checks that the
task has metric fields, sample IDs are unique integers, raw and filtered
responses exist, and the sample count matches the declared count.
For example, a synthetic schema fixture, not a measured result, can have
this result file named results_fixture.json:
{"results":{"gsm8k":{"exact_match,none":0.0}},"config":{"model":"fixture-no-inference"}}
And one line in samples_gsm8k_fixture.jsonl:
{"doc_id":0,"resps":[["fixture answer"]],"filtered_resps":["fixture answer"]}
Run the inspector with expected count 1 for that fixture. It prints the task,
count, metric keys and the two artifact paths. The zero in the fixture is only
placeholder data; the inspector does not recompute or endorse that metric.
The script was exercised on this schema shape and duplicate IDs, absent raw
responses, count mismatches, absent task metrics and ambiguous multi-run
folders, without loading a model or downloading a dataset.
A structural pass is only a preflight. Manually inspect the saved configuration, resolved task/model revisions, prompt and chat template, stop/truncation behavior, per-sample outputs and scoring filters. The checker cannot establish that two files came from the same process, that the expected IDs are the correct dataset split, that an answer is correct, or that the recorded metric matches a recomputation. Preserve those provenance and evaluator checks before claiming a benchmark or comparison. A five-sample smoke remains a smoke even when every artifact check passes.
Source contract: the harness sample writer
and sample-record construction,
release 0.4.10. Recheck this inspector’s assumptions before using another
release. The paired prediction example
is a separate exact-string format; raw harness files cannot be substituted for it.
Interpret failures before interpreting scores
- Readiness or connection failure: check the process, bind host, port and URL.
- A message-list assertion: include
--apply_chat_templateand use the full chat-completions URL. loglikelihoodnot implemented: the task and adapter do not match. Choose a suitable scoring backend with matching tokenizer and log-probability support.- Truncated prompts or incomplete generations: preserve the failure and the context/output limits. Do not compare the result with an uncapped full run.
- Missing artifacts or a nonzero harness exit: inspect server/harness logs. Missing scores remain missing, not zero.
Freeze the full target and regression suite, dataset revision, prompts, decoding settings and acceptance thresholds before a paired comparison. Record latency, RAM and throughput only when measured under that protocol. No wall-clock estimate or expected model score is qualified by this recipe.
Retained HTTP smoke evidence
The historical documentation retained these responses from the separate
posttrainllm-serve-smoke executable and a 27M checkpoint. They show the recorded
HTTP response shapes and stop behavior. They do not qualify the current main
binary, model quality or a completed lm-eval run.
$ curl -s http://127.0.0.1:8765/v1/models
{"object":"list","data":[{"object":"model","id":"posttrainllm","owned_by":"posttrainllm"}]}
$ curl -s -X POST http://127.0.0.1:8765/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model":"posttrainllm","messages":[{"role":"user","content":"Once upon a time"}],"max_tokens":10,"temperature":0.0}'
{"id":"chatcmpl-...","object":"chat.completion",
"choices":[{"message":{"role":"assistant","content":"The first step in the process is to make a"},
"finish_reason":"stop","index":0}],
"model":"posttrainllm","created":...,
"usage":{"prompt_tokens":13,"completion_tokens":10,"total_tokens":23}}
$ curl -s -X POST http://127.0.0.1:8765/v1/completions \
-H "Content-Type: application/json" \
-d '{"prompt":"Once upon a time","max_tokens":15,"temperature":0.0}'
{"object":"text_completion",
"choices":[{"text":" of time, the time of time is not always a matter of time.",
"finish_reason":"stop","index":0}],
"usage":{"prompt_tokens":4,"completion_tokens":15,"total_tokens":19}}
# Stop sequence smoke — generation cuts off before "time" appears.
$ curl -s -X POST http://127.0.0.1:8765/v1/completions \
-H "Content-Type: application/json" \
-d '{"prompt":"A","max_tokens":50,"stop":["time"],"temperature":0.0}'
{"object":"text_completion",
"choices":[{"text":". 1998.\n- \"The first ", "finish_reason":"stop", …}], …}
Source files
native-mac/Sources/TinyGPT/TinyGPT.swift: mainservedispatch.native-mac/Sources/TinyGPTServe/Serve.swift: HTTP handlers, chat formatting and completion scoring.python_ref/lm_eval_tinygpt.py: spawn/reuse and argument forwarding.evals/run_quality_evals.sh: retained driver with incompatible chat-task defaults described above.
For the benchmark landscape and contamination discussion, use the retained benchmark research. Historical research claims do not substitute for qualification on a chosen runtime.