File-ops hard gate
100%up from 58% stock 4BmeasuredQwen3-4B File-Ops Distilled
This is the strongest model win in the repo: a Mac-built specialist reaches 100% on the file-ops hard gate. It is also the clearest example of why routing is mandatory, because breadth drops outside the trained domain.
Headline Numbers
Evidence class applies to each value: measured = retained evaluation or runtime result; historical = recorded result not freshly reproduced; derived = calculated or decided from records; observed = current artifact state; not-measured = an explicit evidence gap.
Heldout file-ops
95%hardgen heldout suitemeasuredBreadth after tuning
42.3%down from 59.6% stockmeasuredArtifact size
7.5GBlocal HF/MLX safetensors directoryobservedCompetitive Context
| System | Metric | Score | Size / Class | Comparable? | Readout |
|---|---|---|---|---|---|
| posttrainllm Qwen3-4B file-ops specialist | local file-ops hard gate | 100% | 4B, 7.5GB package | Direct | Domain specialist result; not a general BFCL leaderboard submission. |
| Stock Qwen3-4B | same local file-ops hard gate | 58% | 4B | Direct | Before/after delta is +42 points on the frozen domain gate. |
| Frontier calibration | same local file-ops hard gate | ~99-100% | frontier API/teacher | Direct | Used as the ceiling check for whether the eval is a usable ruler. |
| BFCL V4 public leader | overall BFCL V4 accuracy | 75.0% | large public model | Directional | LLM Stats snapshot for Qwen3.7 Max; it marks BFCL-V4 rows as self-reported/unverified, so this is market context only. |
| BFCL V4 public average | overall BFCL V4 accuracy | 61.1% | 13 tracked models | Directional | LLM Stats reports 13 self-reported rows and 0 verified rows; posttrainllm still needs a full BFCL submission for direct comparison. |
| Qwen3.5-4B public BFCL-V4 row | overall BFCL V4 accuracy | 50.3% | 4B | Directional | Closest public 4B-class tool-calling row in the same LLM Stats snapshot, but still not the local file-ops gate. |
Direct rows share this artifact's eval setup. Directional rows are useful market context but should not be read as leaderboard claims.
Measured result
| Gate | Stock | Specialist | Readout |
|---|---|---|---|
| File-ops hard gate | 0.58 | 1.00 | Domain win |
| File-ops hardgen heldout | - | 0.95 | Generalizes within file ops |
| Out-of-domain breadth | 0.596 | 0.423 | Regression; route only |
Release Blockers
Breadth regression
The tuned model is wrong to use as a general planner.
Unblock: Keep all public copy routed-only and include the negative-transfer table.
Fine-Tune Report Card
The portable before/after proof for this artifact: baseline, candidate, deltas, regressions, slices, performance, eval validity, and the ship/retry/reject decision — with every value labeled measured, derived, historical, skipped, or not recorded. Compiled offline from the recorded factory evidence; no model was run to produce it.