Public exact
0.53164-row b-mc2/sql-create-context slicemeasuredQwen3-0.6B Routed SQL Specialist
This is the cleanest current proof of the factory thesis: a tiny 0.6B model can beat a small public SQL baseline on a frozen exact-match slice, but only the routed artifact survives both public and execution-style gates.
Headline Numbers
Evidence class applies to each value: measured = retained evaluation or runtime result; historical = recorded result not freshly reproduced; derived = calculated or decided from records; observed = current artifact state; not-measured = an explicit evidence gap.
T5-small baseline
0.484same 64 public rowsmeasuredSynthetic execution
0.86050 heldout SQLite rowsmeasuredSynthetic exact
0.840same 50 heldout rowsmeasuredCompetitive Context
| System | Metric | Score | Size / Class | Comparable? | Readout |
|---|---|---|---|---|---|
| posttrainllm routed SQL v1 | b-mc2 exact / synthetic exec | 0.531 / 0.860 | 0.6B base + 2 routed LoRAs | Direct | Current local candidate; public exact and synthetic execution gates are both frozen. |
| T5-small local baseline | b-mc2 exact | 0.484 | ~60M | Direct | Same 64-row public slice; posttrainllm is +4.7 points exact on this narrow gate. |
| Defog SQLCoder-7B-2 | Defog SQL-Eval category scores | 77.1-96% | 7B | Directional | Strong public SQL specialist, but reported as category-level Defog SQL-Eval scores rather than this b-mc2 slice. |
| Snowflake Arctic-Text2SQL-R1-7B | BIRD execution accuracy | 68.47% | 7B | Not comparable | Useful target class for public SQL execution; posttrainllm must add a BIRD/Spider execution gate before claiming this lane. |
| Snowflake Arctic-Text2SQL-R1-14B / 32B | BIRD execution accuracy | 70.04% / 71.83% | 14B / 32B | Not comparable | Shows the current public high bar: execution accuracy, not exact string match. |
Direct rows share this artifact's eval setup. Directional rows are useful market context but should not be read as leaderboard claims.
Adapter comparison
| Setup | Public exact | Synthetic exec | Decision |
|---|---|---|---|
| Public v4 only | 0.531 | 0.240 | Route required |
| Blend v1 | 0.297 | 0.560 | Reject |
| Best static composition | 0.516 | 0.460 | Reject |
| BIRD + b-mc2 v5 | 0.438 | 0.280 | Reject |
| Classifier-routed v1 | 0.531 | 0.860 | Current best |
Router verification
| Check | Result | Evidence |
|---|---|---|
| Unlabeled mixed rows | 114 | 64 public / 50 synthetic |
| Public route reason | 64 | known_public_source |
| Synthetic route reason | 50 | sqlite_db_field |
| Route confidence | >= 0.99 | all smoke rows |
Output-hygiene DPO retries (composed, 50 heldout rows)
| Run | Execution | Clean-SQL | Decision |
|---|---|---|---|
| Frozen baseline (SFT) | 0.860 | 0.000 | reference |
| 2026-07-04 ref-free SimPO | 0.080 | 0.000 | retry-training (collapse) |
| 2026-07-11 ref-anchored DPO | 0.900 | 0.000 | retry-training |
| 2026-07-11 higher-pressure DPO | 0.920 | 0.000 | retry-data |
Release Blockers
Public execution benchmark missing
b-mc2 exact match is useful, but serious SQL claims need execution accuracy on public DBs.
Unblock: Add BIRD Mini-Dev SQLite or Spider SQLite execution fixtures once the DB bundle is local.
Output hygiene (raw completions carry an Answer:/Explanation: wrapper)
Two reference-anchored DPO retries (2026-07-04 SimPO collapse → fixed; 2026-07-11 gentle + higher-pressure) improved execution 0.860 → 0.900 → 0.920 but left the clean-SQL raw rate at 0.000. The wrapper is a base-model prose prior a rank-4 preference adapter can't strip — the 108 SFT targets are already bare SELECT. Composed DPO is ruled out for hygiene.
Unblock: Generation-strength fix, not more DPO: stronger SFT (higher rank / more examples) or inference-time steering (constrained SELECT-prefix decoding / stop sequence). Execution is not the problem; only the wrapper is.
Not a specialist package yet
The adapters currently live under gitignored run folders, not package metadata.
Unblock: Package under specialists/ only after a ship decision on a public execution gate.
Fine-Tune Report Card
The portable before/after proof for this artifact: baseline, candidate, deltas, regressions, slices, performance, eval validity, and the ship/retry/reject decision — with every value labeled measured, derived, historical, skipped, or not recorded. Compiled offline from the recorded factory evidence; no model was run to produce it.