Every public model, with the result and the failure mode.

OffHours Context-Interference Benchmark

A paired office-work benchmark that separates unresolved personal context from interruption and raw context volume. Devin showed no mental-toll penalty, but reached a reproducible context-volume boundary.

Clean accuracy

99.0%

Mental-toll effect

not detected

Volume boundary

8,000 words/day

Competition: Clean workday vs Resolved family context

Next: Freeze this report-only result. A local provenance-complete comparison, if ever desired, begins as a fresh experiment after the learning phase.

Needle 2: The 45M Capacity Boundary

Catalog restriction and a four-arm successor factorial both failed, establishing that this 45M model class—not one missing recipe tweak—is the current tool-selection bottleneck.

Tiny-overfit gates

4/4

Best successor

26/94

Destructive bypasses

10 vs 2

Competition: Needle 2 full catalog vs Needle 2 oracle task catalog

Next: Close the 45M recipe-search lane. Keep the complete factorial as a hands-on learning artifact; any 1.7B successor is a fresh experiment.

Parakeet WGSL Browser ASR

Parakeet v3 scored 0/82 word errors and ran 3.51x faster than native WhisperKit on the same eight LibriSpeech clips, while missing a deliberately aggressive 50x short-clip speed gate.

Browser WER

0.0%

Native WER

8.54%

Paired latency

3.51x

Competition: Parakeet TDT 0.6B v3 · browser vs WhisperKit Mac-native baseline

Next: Closed as a quality and native-latency win with a formal speed-threshold miss. Use it to teach paired evaluation, fixed-cost amortization, and why frozen gates matter.

Qwen3-0.6B Routed SQL Specialist

A two-adapter routed SQL artifact: public schema-only SQL routes to a b-mc2 adapter; local SQLite execution routes to a synthetic execution adapter.

Public exact

0.531

T5-small baseline

0.484

Synthetic execution

0.860

Competition: posttrainllm routed SQL v1 vs T5-small local baseline

Next: Keep the published report artifact report-only. Two hygiene DPO retries proved reference anchoring cures the SimPO collapse and even lifts execution to 0.920, but output hygiene needs a generation-strength fix (stronger SFT or constrained SELECT-prefix decoding), not more preference tuning. Do not present it as a shipped SQL model until public execution and clean-output gates pass.

Report only — no model to usenot fully verifiedReport card →

Pace Intent Router v8

A 49.5M intent classifier that was extremely fast and strong on its source-matched synthetic holdout, then failed the first leakage-checked sealed distribution gate.

Hugging Face model ↗

Sealed accuracy

57.1%

Frontier retained

57.1%

Warm latency

3.8ms

Competition: Pace Intent Router v8 vs Qwen3-4B-Instruct-2507 4-bit

Next: Do not tune on sealed V1. Build a broader public-development corpus from the failure themes, train a successor, and judge it once on a newly generated sealed V2.

Qwen3-4B File-Ops Distilled

A routed 4B file-operation specialist distilled from frontier/gold trajectories, with the breadth regression disclosed in the package.

Hugging Face model ↗

File-ops hard gate

100%

Heldout file-ops

95%

Breadth after tuning

42.3%

Competition: posttrainllm Qwen3-4B file-ops specialist vs Stock Qwen3-4B

Next: Keep routed-only warnings attached and add a consumer pull/load smoke before wiring this into any app.

Shipped — routed onlynot fully verifiedReport card →

Qwen3-4B ReST Fused

Fresh requalification confirms a perfect, safer, faster file-ops specialist while rejecting it as the general successor because breadth regressed.

Hugging Face model ↗

File-ops hard gate

12/12

Qualified breadth

25/45

Depth wall speedup

2.42x

Competition: posttrainllm Qwen3-4B ReST vs Stock Qwen3-4B

Next: Keep this package behind the file-ops route and reject it as the general Pace successor.

Shipped — routed onlynot fully verifiedReport card →

Qwen3-4B Multibackend Distilled

A public failed attempt that retained 100% file-ops depth but drove the recorded breadth score down to 31%.

Hugging Face model ↗

File-ops depth

100%

Breadth

31%

Breadth delta

-28.6pp

Competition: Multibackend-distilled 4B vs Stock Qwen3-4B

Next: Keep the weights public as a failed comparison artifact. Do not spend compute revalidating them unless a new breadth-preserving recipe explicitly needs this checkpoint as its baseline.

VibeThinker-3B MLX Conversion

A public Apple-Silicon-friendly conversion of WeiboAI/VibeThinker-3B, preserved as a runtime artifact rather than claimed as a PostTrainLLM-trained model.

Hugging Face model ↗

Model class

3B

Local GSM8K screen

40/40

Training delta

None

Competition: PostTrainLLM VibeThinker-3B MLX vs WeiboAI/VibeThinker-3B

Next: Keep this as a conversion and runtime artifact. Add a pinned loader plus parity receipt only when a real Mac-local consumer justifies maintaining it.

VibeThinker-3B Agentic Distilled

A preserved agentic distillation checkpoint whose public weights outlived its evaluation evidence; no current win can be claimed.

Hugging Face model ↗

Public weights

Yes

Current agentic eval

Missing

Reasoning retention

Unknown

Competition: VibeThinker-3B agentic distilled vs VibeThinker-3B MLX base

Next: Leave the weights public but unpromoted. If this lineage becomes active again, start with an evidence-only baseline/candidate evaluation—not more training.

Hugging Face Specialist Model Archive v1

The local specialist model cache was promoted to Hugging Face or deleted when safely re-downloadable from upstream repos.

posttrainllm HF repos

6

Local model cache

cleared

Storage policy

HF first

Competition: Hugging Face artifact storage vs Local Mac cache

Next: Use this as the storage index; use the six dedicated model case studies for quality and decision evidence.

Factory Run Schema v1

The canonical target, data, post-training, evaluation, package, and report shape for PostTrainLLM runs.

Required files

8

Decisions

6

First-class outputs

5

Competition: posttrainllm factory schema vs Ad hoc model card only

Next: Use the routed SQL report as the canonical example; make live commands emit the same complete contract automatically.

Browser WebGPU Training Speedup

A frozen ABBA run on an Apple M5 Pro measured a 10.67x Large-preset WebGPU speedup over WASM with 4.72% paired final-loss drift and zero runtime errors.

WebGPU speedup

10.67x

Loss drift

4.72%

Hardware gate

pass

Competition: posttrainllm WebGPU vs posttrainllm WASM SIMD

Next: Keep Large as the measured M5 Pro win; reproduce Small, Medium, and XL independently before promoting the historical curve.

Memory64 Browser Behemoth Allocation

A WebAssembly Memory64 build lifted the browser model allocation ceiling past the old 4GB tab limit.

Allocated params

473M

Allocation time

3.7s

Train step

82.2s

Competition: posttrainllm Memory64 build vs posttrainllm wasm32 build

Next: Keep this as a public technical artifact; do not make it active factory work unless a browser-run specialist needs it.

Huge Preset Decode Throughput

The native Mac runtime reached high local decode throughput on the Huge preset, showing the serving path is viable for local eval loops.

Huge decode

696 tok/s

Mega pilot

293 tok/s

Warm TTFT p99

5.8ms

Competition: posttrainllm Huge preset vs posttrainllm Mega pilot

Next: Use this as the baseline expectation for future artifact performance tables.