Train a small GPT — the kind of model behind ChatGPT, a million times smaller — from scratch, right here in this tab. No server, no install. Watch the loss curve fall, then ask it to write you a sentence. Source ↗
Train
Hyperparameters — preset above sets these. Click to edit individually.
Advanced
Loss curve wasm system: nominal
More metrics
Sample live
You can sample while training is still running — this is the trick. The model is a single instance in a Web Worker; Generate runs a forward pass against the same weights training is mutating, between steps. Sample twice during a run to watch what the model has picked up.
Inspect & evaluate
Benchmark
Score the loaded model against a registered held-out set.
Logit lens interpretability
Per-layer predictions for the current prompt. Reveals when specific knowledge appears in the residual stream. Optionally upload a tuned-lens .lenses sidecar trained via "posttrainllm tuned-lens" for calibrated probes.
Ablation interpretability
Zero out one layer's attention, MLP, or whole-block contribution, then sample. Reveals how load-bearing that component is.
Activation patching interpretability
Intervene on the residual stream at (layer, position) of the recipient prompt above. Zero: drop that token's representation at that depth — exposes whether it's load-bearing. Swap: substitute a hidden state captured from a separate donor prompt at the same coords — the full Meng et al. (2022) variant that reveals which downstream outputs causally depend on the donor's representation.
Watch the model think
Click any byte below to see what the model considered at that
position — the top-10 next-byte probabilities the
last forward pass produced, and the per-head attention
weights from the final transformer block (which earlier
bytes each head was looking at).
This is honest: it's the actual distribution that token was sampled
from, and the actual softmax weights from the last attention layer.
Nothing prettified. · marks bytes that are whitespace
or control characters.
Top-10 candidates
Attention from last layer · per head
Your machine detecting… computing…
computing…
4 Resources curated rabbit holes — 3Blue1Brown, Karpathy, papers, and this repo's own docs
Start here (visual + intuitive)
- 3Blue1Brown — Neural Networks — the maths, visually
- Jay Alammar — The Illustrated Transformer — the diagrams everyone has seen
- Karpathy — Let's build GPT — a 2 h build, from zero
- Karpathy — Neural Networks: Zero to Hero — the full lecture series
Go deeper (code + papers)
- karpathy/nanoGPT — the cleanest small GPT in PyTorch
- karpathy/minGPT — the original minimal build
- Attention Is All You Need — Vaswani et al., 2017
- LoRA: Low-Rank Adaptation — Hu et al., 2021
- AdamW — Loshchilov & Hutter, 2017
This project's own write-ups
- docs/notes.md — how every layer was built & tested
- docs/learn/README.md — the active learning index
- docs/techniques/lora_guide.md — LoRA in detail
- docs/performance/performance.md — the SIMD & WebGPU speed work
- docs/browser_notes.md — WASM, Workers, OPFS, WebGPU
Related projects
- mlc-ai/web-llm — run real LLMs in the browser (inference)
- ml-explore/mlx — Apple Silicon ML framework
- Hugging Face datasets — where the corpus picker pulls from
- MDN — WebGPU — the API powering the GPU path
5 WebGPU matmul benchmark Same matmul on WebGPU + WASM, side-by-side, with a live speedup number
Runs the same matmul through the WebGPU compute kernel and the WASM kernel, parity-checks they agree, and reports the per-shape speedup on your hardware. Needs Chrome / Edge 113+. For the historical end-to-end curve and its evidence limits, see /speedup.