Docs Quality Audit
This page tracks whether posttrainllm documentation is actually world-class, not just large.
Definition
Docs are world-class when an outsider can answer:
- What is the project?
- What is active vs parked?
- What did we try?
- What worked, failed, or regressed?
- What products/papers/blogs changed the plan?
- What is on the roadmap?
- What should the owner learn next?
- How is a public claim validated?
- How can the next run reproduce or improve the result?
Current State
| Requirement | Current Source | Status |
|---|---|---|
| Project state | PROJECT_STATUS.md | strong |
| Active roadmap | docs/NEXT.md | strong |
| Golden path | docs/README.md | strong |
| Factory pipeline | docs/factory/ | strong |
| Method vs recipe | docs/techniques/method-vs-recipe.md | strong |
| Technique audit inventory | docs/techniques/audit-inventory.md | strong |
| Attempt history | docs/attempt-ledger.md, docs/attempts.json, docs/history-coverage-audit.md, scripts/check_attempt_ledger.py | strong |
| Exactness completion audit | docs/exactness-completion-audit.md | complete |
| External products reviewed | docs/external-products-reviewed.md | good, should grow with every teardown |
| Learning pipeline | docs/learning-pipeline.md | strong, tied to current factory work |
| Learning progress | docs/learning-progress.md | good, manually maintained |
| Public artifacts | docs/factory/public-artifacts.md | strong |
| Enforcement | docs/factory/enforcement.md, posttrainllm factory-run publish-check, scripts/check_factory_run_publish.py | strong |
| Docs completeness check | scripts/check_docs_world_class.py | good, checks golden-path surfaces exist |
| Old docs status | docs/README.md, docs/MAP.md, docs/parked/ | acceptable, still noisy |
What Improved In This Pass
- Added a canonical docs entrypoint.
- Added an attempt ledger with worked, failed, regressed, and not-tried attempts plus confidence labels, evidence sources, and a structured sync check.
- Added a history coverage audit so normalized, classified, partial, and narrative-only historical surfaces are separated honestly.
- Added an external products/research review ledger.
- Added a learning pipeline tied to the factory loop.
- Added a method-vs-recipe registry and SQL technique backlog.
- Added a row-level technique inventory for
docs/audit_2026.md, so broad technique rows are classified without being misrepresented as run attempts. - Added an exactness completion audit that records the proof set and non-blocking future hardening.
- Added factory enforcement docs.
- Added a stricter publish-check script.
- Added a docs golden-path completeness check.
- Updated the SQL factory renderer to emit
slice-metrics.jsonandtrace_review.md. - Wired the new docs into
PROJECT_STATUS.md,docs/NEXT.md,docs/MAP.md,docs/factory/README.md,docs/learn/README.md, anddocs/techniques/README.md.
Completion Audit
The current docs meet the world-class baseline defined above:
| Question | Evidence |
|---|---|
| What is the project? | PROJECT_STATUS.md, docs/README.md |
| What is active vs parked? | docs/doc-status.md, docs/NEXT.md, docs/parked/ |
| What did we try? | docs/attempt-ledger.md, docs/attempts.json, docs/history-coverage-audit.md, docs/exactness-completion-audit.md |
| What worked, failed, or regressed? | docs/attempt-ledger.md, docs/techniques/sql-technique-backlog.md, docs/techniques/audit-inventory.md, docs/history-coverage-audit.md |
| What products/papers/blogs changed the plan? | docs/external-products-reviewed.md, docs/techniques/trainloop-teardown.md |
| What is on the roadmap? | docs/NEXT.md, docs/techniques/sql-technique-backlog.md, docs/factory/public-artifacts.md |
| What should the owner learn next? | docs/learning-pipeline.md, docs/learning-progress.md |
| How is a public claim validated? | docs/factory/enforcement.md, posttrainllm factory-run publish-check, scripts/check_factory_run_publish.py |
| How can the next run reproduce or improve the result? | docs/factory/run-schema.md, provenance.json, docs/techniques/ |
Verification commands:
bash evals/docs-world-class-smoke.sh
bash evals/attempt-ledger-smoke.sh
bash evals/technique-inventory-smoke.sh
bash evals/factory-publish-check-smoke.sh
npm run build # from browser/Future Hardening
These are not blockers to the world-class baseline, but they should continue to improve:
-
Attempt ledger still has manual source metadata. It now has
docs/attempts.jsonand a checker, but future run commands should append attempts automatically. -
Publish validation has two implementations.
posttrainllm factory-run publish-checkis the canonical command, whilescripts/check_factory_run_publish.pyremains as a portable smoke. This is acceptable, but should eventually share one implementation. -
Old docs are still noisy, but bounded.
docs/PLAN.md, older PRDs, and session notes remain useful but can distract readers.docs/history-coverage-audit.mdnow names which older surfaces are normalized, classified, or narrative-only. -
External teardown corpus is shallow. TrainLoop, Baseten, SQL specialists, Apple, Castform, and agent-system notes are captured, but every new target should get a fresh competitor/literature teardown.
-
Learning progress is manual. The owner learning sequence now has a progress tracker, but it is not derived from run artifacts.
-
Current SQL report is still report-ready, not ship-ready. Public execution benchmark and routed performance numbers remain blockers.
Next Quality Gates
To keep improving the docs after the baseline:
- Add structured run metadata fields for git commit, binary provenance, dataset hash, and exact commands.
- Add visible
active/reference/archive/stale/supersededbanners to remaining old docs beyond the major entrypoints. - Make
docs/learning-progress.mdupdate from run metadata or checklist commands.