The factory, end to end

Build the ruler.
Then build the model.

A small specialist earns its place one artifact at a time. This is the shortest path through the system—from choosing a capability to deciding whether the result should ship, redirect, retry, or stop.

Start building Open the complete docs

01target
02data
03post-train
04evaluate
05package
06report

Field guides

nine stages / thirteen things you can make

Learn by leaving artifacts behind.

Theory becomes useful when you can change a real system and prove what the change did. Each stage ends in something inspectable: a model, adapter, benchmark, agent, package, runtime decision, or causal probe.

runnable lab

A bounded local or browser exercise with an existing implementation.

guided replay

A source-linked reconstruction of retained evidence; it does not claim a fresh run.

recipe contract

A complete procedure whose expensive execution begins only as a fresh, budgeted experiment.

How does text become a prediction and a gradient?

Make a language model

Trace the complete transformer learning loop, then change it without losing track of tensor shapes.

Open the learning path →

model checkpoint

Byte-level TinyGPT

runnable lab

A readable ~0.8M-parameter transformer whose byte tokenizer, architecture, loss, and checkpoint format are all inspectable.

workloadlight first; training duration is user-controlled

Build: Train the tiny preset on the bundled corpus. →
Modify: Change layers, heads, width, or context in the source-of-truth model config. →
Tune: Change learning rate, batch size, seed, or step budget one variable at a time. →
Prove: Inspect initial loss, tiny-overfit behavior, samples, and tensor inventory. →
Package: Keep a validated .tinygpt checkpoint with its exact config and corpus. →

Inspectmodel.py ↗train.py ↗tiny-corpus.txt ↗

Runposttrainllm trainposttrainllm inspectposttrainllm validate

interactive model lab

Browser-trained TinyGPT

runnable lab

The same learning loop made visible through loss, sampling, presets, workers, WASM, and WebGPU-capable browser surfaces.

workloadruns in the browser; preset size controls cost

Build: Open the playground and train a tiny model without a server. →
Modify: Replace the corpus or choose a different model preset. →
Tune: Adjust context, batch, learning rate, and generation controls in the lab. →
Prove: Watch loss and samples together; do not mistake lower loss for useful behavior. →
Package: Export the browser artifact and retain the settings that produced it. →

Inspectplayground.astro ↗gpu_model.ts ↗model.cpp ↗

Is the learner correct before it is large?

Make training trustworthy

Use tiny-overfit, held-out loss, numerical checks, and bounded comparisons to separate implementation bugs from optimization problems.

Open the learning path →

training evidence

Tiny-overfit correctness receipt

runnable lab

A minimal proof that data loading, forward pass, backpropagation, optimizer state, sampling, and checkpoint reload agree.

workloadbounded correctness gate

Build: Run the smallest overfit gate before any ordinary training run. →
Modify: Change only one of corpus, seed, precision, or optimizer settings. →
Tune: Use learning-rate and clipping changes only after gradients and expected initial loss are verified. →
Prove: Require memorization, checkpoint round-trip, and a sensible held-out comparison. →
Package: Retain the config, seed, loss trace, sample, and validated checkpoint as one receipt. →

Inspectevaluate.py ↗checkpoint.py ↗

Runposttrainllm trainposttrainllm evalposttrainllm debug-lossposttrainllm validate

Which intervention teaches the missing behavior with the least collateral damage?

Change model behavior

Turn data and an adaptation method into an adapter whose target gain and regressions are explicit.

Open the learning path →

adapter

LoRA specialist adapter

recipe contract

A narrow adapter built from a frozen base, traceable data, explicit LoRA geometry, and a held-out task gate.

workloadfresh training requires an explicit budget

Build: Prepare supervised rows and run the retained SFT path against a frozen base. →
Modify: Change the target data or modules while keeping the baseline and holdout frozen. →
Tune: Vary rank, alpha, target modules, learning rate, or steps from the checked starting profiles. →
Prove: Compare target gain, breadth, safety, latency, RAM, and throughput to the unchanged base. →
Package: Bind adapter, base identity, prompt contract, recipe, and evidence into a specialist package. →

InspectSFT.swift ↗registry.json ↗

Runposttrainllm prep-dataposttrainllm sftposttrainllm eval-gateposttrainllm export-mlx

routed specialist

Distilled tool-calling specialist

guided replay

A real lineage where narrow file-operation depth reached 100% while breadth regressed, proving why routing is part of the artifact.

workloadreplay is read-only; reproduction is a budgeted experiment

Build: Reconstruct the student, teacher trajectories, filtering, and training lineage from retained evidence. →
Modify: Change teacher mix or selection slices as a new recipe version, never silently. →
Tune: Tune accepted rows, sampling mix, and steps against the same narrow and breadth gates. →
Prove: Read target and breadth scores together: 100% narrow depth did not make a general upgrade. →
Package: Package as routed-only with an explicit redirect boundary. →

Inspectattempt-ledger.md ↗artifacts.ts ↗

Runposttrainllm distillposttrainllm eval-bfclposttrainllm export-mlx

Does the benchmark measure the capability it claims to measure?

Build the ruler

Freeze a baseline, validate the frontier ceiling, inspect slices, and end with a reproducible decision rather than a score screenshot.

Open the learning path →

benchmark case file

Needle 2 selection ruler

guided replay

A complete tiny-model journey from catalog ablation through a 2x2 distractor-and-safety training factorial, ending at a measured 45M capacity boundary.

workloadpublic evidence replay; rerunning model evaluation requires approval

Build: Build the four training arms and verify the tiny-overfit correctness gate. →
Modify: Inspect how distractor-rich and safety examples differ while every other training variable stays fixed. →
Tune: Compare the four LoRA arms, three independent training seeds, and the predeclared unsafe-arm stop rule. →
Prove: Reproduce the public-dev decision and explain why sealed V2 and quantized parity stayed locked. →
Package: Retain fixtures, raw outputs, scorer, catalog policy, and conclusion as a benchmark artifact. →

Inspectattempts.json ↗needle2-baseline-review.md ↗needle2-successor-factorial.md ↗needle-successor-result-v1.json ↗

Runposttrainllm eval-bfclposttrainllm eval-compare

decision artifact

Specialist factory report card

runnable lab

A run directory that binds target, data, candidate, frozen evaluation, packaging evidence, resource measurements, and a terminal ship/retry/reject decision.

workloadmetadata lifecycle is light; model evaluation may require approval

Build: Create a factory run and populate its declared evidence contract. →
Modify: Version targets, gates, and policies without rewriting completed evidence. →
Tune: Change thresholds only before the candidate is scored, with rationale recorded. →
Prove: Run the same frozen evaluator on baseline and candidate, including regression slices. →
Package: Compile the complete run into a public before/after report and terminal decision. →

InspectFactoryRunCommand.swift ↗build_fine_tune_report_card.py ↗

Runposttrainllm factory-runposttrainllm eval-gateposttrainllm eval-compare

What turns model output into a safe, useful local agent?

Make the model act

Separate model capability, constrained output, validation, tool policy, state, and serving performance in one trace.

Open the learning path →

agent runtime

Local tool-using agent

runnable lab

An OpenAI-compatible local model server plus a multi-turn tool loop with schema validation and explicit execution policy.

workloadlocal inference; model size controls RAM and latency

Build: Serve a local artifact, then run one bounded tool-using conversation. →
Modify: Change tool schemas, catalog routing, prompts, or execution policy independently. →
Tune: Tune decoding, prompt reuse, constraints, and routing before assuming more training is needed. →
Prove: Score call selection, arguments, legality, task completion, latency, and unsafe actions. →
Package: Ship the model, tool catalog, policy, and eval gate as one bounded system contract. →

InspectAgentLoop.swift ↗Serve.swift ↗

Runposttrainllm serveposttrainllm agentposttrainllm eval-bfclposttrainllm eval-tau-bench

Where do the time, memory traffic, and numerical drift actually come from?

Understand the machine

Use the readable implementation as an oracle while comparing kernels and architectures with end-to-end measurements.

Open the learning path →

systems benchmark

Kernel parity and speed receipt

guided replay

A numerical and performance comparison that keeps reference math, optimized kernel, dispatch geometry, and end-to-end effect separate.

workloadread-only replay; fresh kernel loops require approval

Build: Trace reference attention and matmul before inspecting their optimized counterparts. →
Modify: Change precision, tile size, workgroup shape, or architecture one dimension at a time. →
Tune: Tune the measured bottleneck rather than optimizing a convenient microbenchmark. →
Prove: Require numerical parity, warmed timing, memory evidence, and end-to-end impact. →
Package: Retain source revision, device, shapes, precision, raw timings, and parity tolerance. →

Inspectmodel.py ↗attention_fa2.wgsl ↗fa2_backward_notes.md ↗

Runposttrainllm benchposttrainllm bench-trainposttrainllm experimental train-heads

Did the artifact become cheaper to store, load, and run without losing its job?

Make it smaller and portable

Distinguish file compression, RAM reduction, fewer operations, runtime kernel support, and retained capability.

Open the learning path →

deployable model bundle

Validated MLX specialist package

runnable lab

A model or adapter exported with identity, tensors, tokenizer, prompt contract, and evidence that the consumer can load it correctly.

workloadconversion is bounded; validation loads the selected model

Build: Export a .tinygpt model, adapter, or Hugging Face directory into the MLX layout. →
Modify: Change the packaging boundary or consumer metadata without changing model claims. →
Tune: Choose quantization or pruning only against frozen quality, RAM, size, and speed gates. →
Prove: Verify round-trip load parity, file size, peak RAM, tok/s, and target quality. →
Package: Publish the bundle with model card, hashes, environment, limits, and decision. →

InspectExportMLX.swift ↗TinyGPT.swift ↗

Runposttrainllm export-mlxposttrainllm validateposttrainllm experimental gptqposttrainllm experimental prune-structured

Which parts belong in Python, Swift and MLX, Core ML, WASM, or WebGPU?

Cross runtime boundaries

Choose a runtime from capability, ownership, memory, latency, energy, distribution, and browser constraints—not platform novelty.

Open the learning path →

browser inference artifact

Parakeet browser ASR receipt

guided replay

A paired browser/native ASR lab: Parakeet wins bounded WER and latency while a frozen 50x short-clip throughput gate catches fixed dispatch cost.

workloadeight reference-scored LibriSpeech clips against native WhisperKit

Build: Trace model download, cache, WebGPU load, audio preprocessing, and decoded transcript. →
Modify: Change audio input, caching policy, or runtime adapter while preserving the receipt schema. →
Tune: Tune chunking, preprocessing, and execution policy before changing model weights. →
Prove: Recompute paired WER, proper nouns, repetitions, per-clip RTFx, and the frozen decision. →
Package: Retain runtime versions, browser/device facts, cache state, input provenance, output, and limitations. →

Inspectattempts.json ↗artifacts.ts ↗

architecture decision record

Mac runtime boundary map

guided replay

A comparison of owned MLX weights, Core ML deployment, Apple Foundation Models, and browser runtimes grounded in measured capability limits.

workloadread-only evidence synthesis

Build: Map one use case across capability, context, RAM, energy, ownership, and distribution constraints. →
Modify: Add a new runtime only with a directly comparable evidence row. →
Tune: Tune routing and deployment policy; do not hide a capability gap behind serving changes. →
Prove: Use the same task gate plus latency, RAM, tok/s, energy, setup, and ownership constraints. →
Package: Record the chosen runtime, rejected alternatives, evidence, and reversal conditions. →

Inspectfm_agent_bridge.swift ↗CoreMLServe.swift ↗

Runposttrainllm hf-loadposttrainllm ane-validateposttrainllm coreml-serve

Which internal representation causally contributes to a measured behavior?

Look inside without fooling yourself

Turn a behavioral failure into a probe, control, intervention, and decision-changing prediction.

Open the learning path →

interpretability artifact

Causal probe dossier

recipe contract

A logit lens, linear probe, activation patch, causal trace, or sparse autoencoder investigation with negative controls and an intervention claim.

workloadattach to a specific failure before running

Build: Start from frozen success and failure traces plus a falsifiable representation hypothesis. →
Modify: Change layer, token position, feature, or control while keeping the behavioral gate fixed. →
Tune: Tune probe or SAE settings on train data; reserve separate examples for the causal claim. →
Prove: Require random controls, replay stability, and a predicted behavior change under intervention. →
Package: Retain activations or hashes, probe weights, controls, intervention results, limitations, and recipe decision. →

InspectExperimentalCommands.swift ↗session-10-attention.md ↗

Runposttrainllm experimental tuned-lensposttrainllm experimental linear-probeposttrainllm experimental causal-traceposttrainllm experimental sae

learning lab / nine ready paths

Learn it, inspect it, then run it.

Each path has prerequisites, a repository lab, and a mastery gate. Start with foundations; use the others when their prerequisites are earned.

start here

Foundations to transformer

Functions, data, and parameters
Loss and gradient descent
Vectors, matrices, and tensors
Non-linear networks and backpropagation
Tokenization and embeddings
Attention and transformer blocks

Lab: Complete the toy calculation or implementation in Modules 1-7 and explain its tensor shapes aloud.
Mastery gate: Trace text from tokens through embeddings, attention, MLP, logits, loss, and one gradient update without skipping a transformation.
CLI entry points: posttrainllm trainposttrainllm inspect

Open the path →Foundation prerequisite; applied recipes begin in later paths.

7 evidence anchors

after foundations

Training mechanics and stability

Batches and masking
Optimizers and schedules
Precision and numerical checks
Tiny-overfit correctness gate
Memory and convergence diagnosis

Lab: Inspect one successful tiny-overfit and one failed ordinary run, then diagnose data, optimization, capacity, and evaluation separately.
Mastery gate: Given a loss curve and held-out result, identify what is proven and choose the smallest discriminating next test.
CLI entry points: posttrainllm trainposttrainllm bench-trainposttrainllm eval

Open the path →Recipe: GaLore and training stability →Recipe: Optimizer and schedule selection →

5 evidence anchors

after training-mechanics

SFT, adapters, preference tuning, and distillation

SFT behavior learning
LoRA geometry
DPO and reference anchoring
Distillation
Routing and breadth retention

Lab: Reconstruct the SQL and file-ops lineages, including the variable changed, target gain, regression, and decision at each step.
Mastery gate: Design a recipe with data, target, reward or labels, regression slices, resource budget, stop rule, and ship/retry/reject threshold.
CLI entry points: posttrainllm sftposttrainllm dpoposttrainllm distillposttrainllm finetune

Open the path →Recipe: Specialist distillation →Recipe: Evolution strategies →Recipe: LoRA, DoRA, and adapter geometry →Recipe: Parameter-efficient fine-tuning variants →

6 evidence anchors

after post-training

Evaluation, evidence, and the specialist factory

Frontier-ceiling validation
Frozen baselines and heldouts
Slice and safety gates
Trace review
Packaging and report cards
Ship, route, retry, or reject

Lab: Audit Needle, Chess, and 2048 and explain how the ruler or safety gate changed the decision before integration or training.
Mastery gate: Reject a superficially good score when its frontier ruler, safety slice, breadth gate, or provenance is invalid.
CLI entry points: posttrainllm factory-runposttrainllm eval-gateposttrainllm eval-compareposttrainllm inspect

Open the path →Recipe: SQL specialist lineage →Recipe: Needle 45M successor factorial →

6 evidence anchors

after foundations + evaluation-and-factory

Inference, serving, tools, and long context

Autoregressive decoding
KV cache and prompt reuse
Constrained generation
Tool dispatch and routing
Speculative decoding
Long-context cache policies

Lab: Calculate KV memory, compare cold versus reused-prefill latency, and trace one tool request through model output, validation, and execution policy.
Mastery gate: Separate model capability, serving optimization, validation, and agent policy in one end-to-end trace.
CLI entry points: posttrainllm sampleposttrainllm serveposttrainllm agentposttrainllm eval-bfcl

Open the path →Recipe: Constrained generation →Recipe: Multi-token prediction →Recipe: Speculative heads and decoding →Recipe: StreamingLLM and KIVI cache compression →

7 evidence anchors

after foundations + training-mechanics

Architecture and kernel execution

Reference transformer math
Attention memory traffic
Precision and tiling
Sparse versus dense MoE
Correctness before performance

Lab: Match one optimized attention or matmul path against the readable reference and identify the real bottleneck with a measured curve.
Mastery gate: Explain the numerical contract, memory traffic, and why a microbenchmark may not improve end-to-end training.
CLI entry points: posttrainllm benchposttrainllm bench-trainposttrainllm experimental train-heads

Open the path →Recipe: Sparse mixture of experts →Recipe: Precision and numerical gates →

6 evidence anchors

after training-mechanics + evaluation-and-factory

Quantization, pruning, and artifact packaging

Calibration and numeric formats
Storage versus runtime kernels
Pruning and physical topology
Load and serve parity
Model cards and public evidence

Lab: Audit one int4 or pruned artifact from source weights through file size, load parity, quality, RAM, and runtime measurement.
Mastery gate: Distinguish a smaller file, lower RAM, fewer physical operations, and faster wall-clock execution.
CLI entry points: posttrainllm export-mlxposttrainllm validateposttrainllm experimental gptqposttrainllm experimental prune-structured

Open the path →Recipe: Unstructured and structured pruning →Recipe: Quantization and storage formats →

5 evidence anchors

after architecture-and-kernels + runtime-and-agents

Browser, WASM, WebGPU, MLX, and Apple runtime

WASM memory and SIMD
WebGPU device and kernel model
Browser caching and distribution
MLX unified memory
Apple Foundation Models and Core ML boundaries
Browser ASR case study

Lab: Compare the historical unqualified WebGPU curve and its missing receipt, Memory64 limit, Apple action-grounding result, and Parakeet browser receipt as four different capability and evidence boundaries.
Mastery gate: Choose the correct Mac or browser runtime from capability, memory, latency, energy, download, and ownership constraints.
CLI entry points: posttrainllm hf-loadposttrainllm ane-validateposttrainllm coreml-serveposttrainllm vlm-smoke

Open the path →Recipe: Parakeet WGSL browser ASR →

6 evidence anchors

after foundations + evaluation-and-factory

Interpretability with causal discipline

Attention and logit inspection
Linear probes
Activation patching and causal tracing
Sparse autoencoders
Controls and intervention claims

Lab: Take one behavioral failure and write a probe hypothesis, negative control, intervention, and decision-changing prediction.
Mastery gate: Refuse to turn an attractive visualization into a causal model claim without an intervention.
CLI entry points: posttrainllm experimental tuned-lensposttrainllm experimental linear-probeposttrainllm experimental causal-traceposttrainllm experimental sae

Open the path →Recipe: Interpretability probes →

3 evidence anchors

00 / admission rule

Frontier is the calibration anchor, not the opponent.

If a strong frontier model cannot nearly ace a benchmark, that benchmark cannot grade the Mac model. Fix or drop the ruler first. The objective is frontier capability retained per unit of model size, RAM, latency, and cost.

01–06 / the retained lab loop

Six questions. Six durable artifacts.

The narrative is the index. The linked docs and code remain the source of truth.

Can a narrow capability be specified tightly enough to measure?

Choose a target worth compressing.

Start with the task, the incumbent, and a frozen success bar—not a model. A frontier model first proves the ruler can represent the capability at all.

emitstarget.json + frozen baseline

ReadFactory contract →Method vs recipe →

SourceFactoryRunCommand.swift ↗EvalGate.swift ↗

Which decisions are missing, and where did each training row come from?

Build data for the failure, not the benchmark name.

Trace review turns misses into slices. Synthetic teachers are useful only when provenance, deduplication, contamination checks, and a held-out gate stay attached to the rows.

emitsdataset + manifest + provenance

ReadDataset inventory →Distillation recipe →

SourceDatasetRegistry.swift ↗SFTCorpus.swift ↗

Does SFT, preference training, or rejection sampling improve the frozen target?

Change the smallest model that can plausibly close the gap.

The readable Swift + MLX path remains the local reference. Every intervention is a recipe with exact data, adapter, optimizer, steps, and composition—not just a technique name.

emitsadapter + recipe + training trace

ReadPost-training factory →Tool-calling case study →

SourceSFT.swift ↗FactoryRunLifecycle.swift ↗

Did capability rise on the frozen target without unacceptable regressions?

Make the candidate earn the claim.

The same evaluator scores baseline and candidate. Slice metrics, legality, raw outputs, latency, RAM, throughput, and uncertainty stay visible; a benchmark whose frontier ceiling fails is rejected.

emitsmetrics + traces + regression gate

ReadEvaluation protocol →Benchmark archive →

SourceEvalCompare.swift ↗EvalBFCL.swift ↗

Where is this specialist allowed to answer, and when must it redirect?

Package the boundary, not just the weights.

A specialist artifact includes identity, base model, adapter, prompt contract, eval evidence, resource measurements, and routing constraints. Narrow depth can ship only with an honest boundary.

emitsvalidated specialist package

ReadPackaging contract →Public specialists →

SourceFactoryRun.swift ↗FineTuneReportCard.swift ↗

Should this candidate ship, retry, redirect, or be rejected?

End every run with a decision.

The report binds score delta to regressions, cost, latency, RAM, throughput, missing measurements, and source hashes. Failed attempts remain part of the system because they change what gets tried next.

emitsreport card + ship / retry / reject

ReadReporting contract →Attempt ledger →

Sourcebuild_fine_tune_report_card.py ↗factory_run_lifecycle.py ↗

case files / decisions, not demos

What the loop has actually taught us.

An improvement, a regression, and a failed ruler are all useful when they force a clear next decision.

improved

Tool calling: training helps a weak base

The 1.7B arc moved from stock to SFT to GRPO. SFT did most of the work; GRPO added two points, then the approach plateaued below the stronger 4B.

Read the full arc →

56 → 76 → 78

routed

File ops: depth won, breadth regressed

The distilled 4B aced the narrow hard gate while losing out-of-domain breadth. That is a routed specialist result, not a general model upgrade.

Inspect the package →

58 → 100 / 59.6 → 42.3

rejected ruler

Character 2048: no intelligence gradient

The strongest completed frontier attempt did not clear the frozen 1.10× bar over paired random legal play. Training was stopped and the replay was retained.

Watch the failed attempt →

1.058× best frontier ratio

candidate

Character Chess: promising, not frozen

The first suite showed a real gradient, but the frontier did not approach the required ceiling. The expanded candidate pool remains for human review—not small-model grading.

Inspect candidate evidence →

65% frontier development score

annotated source / where to begin

Read the system in the same order it runs.

Qwen Chess made its implementation approachable by naming the modules behind each lesson. These are the equivalent entrypoints here; the repository contains the complete implementation and history.

01
FactoryRunCommand.swift ↗
CLI composition and the top-level run boundary.
02
FactoryRunLifecycle.swift ↗
State transitions, locks, manifests, and durable run identity.
03
SFT.swift ↗
The readable Mac-local supervised fine-tuning path.
04
EvalCompare.swift ↗
Baseline-versus-candidate comparison under one contract.
05
build_fine_tune_report_card.py ↗
Offline compiler from validated evidence to the public before/after report.
06
attempt-ledger.md →
The durable memory of shipped, retried, rejected, and incomplete attempts.