T TinyGPT

Open-source transformer lab · 2.6× → 12.1× faster on WebGPU

TinyGPT trains a GPT-2 from scratch in your browser tab.

Hand-written WebGPU kernels run 2.6× → 12.1× faster than the multi-thread WASM SIMD baseline as d_model climbs from 96 to 256. Parity-tested to within 2.5% loss drift. Every modern technique — MoE, MTP, ALiBi, differential attention, distillation, DoRA, StreamingLLM — ships as a single clean Swift or TypeScript file, with a paper link next to it.

Train a transformer in your browser WebGPU · ~15 min to readable pseudo-Shakespeare on M-series + Chrome 113+ Build the Mac CLI & app MLX-Swift · trains up to 1.3B params on unified memory

Open source under MIT. Source: github/tinygpt. Single-machine roadmap: 7 parts, 10 shipping phases.

Architecture menu

Every block is one file. The CLI accepts each as an opt-in flag (--moe-experts, --mtp-horizons, --diff-attn, --yoco, …) so configurations compose — you can train a Huge model with MoE and MTP and differential attention simultaneously, validated in a single shipped run.

Attention variants

Plug-in alternatives to standard causal MHA.

Sparsity & scale

Capacity per byte of memory.

Training

Pre-train, instruction-tune, prefer.

  • Pre-train on raw text (byte-level or BPE via any HuggingFace tokenizer)
  • SFT with response-only loss masking (ChatML / Alpaca / Llama / plain)
  • DPO + SimPO / ORPO / KTO — preference variants in one trainer (DPO, SimPO, ORPO, KTO)
  • Distillation — KL + NLL mix, temperature, alpha (Hinton et al., 2015)
  • Evolution Strategies — gradient-free, antithetic sampling (Salimans et al., 2017)
  • LoRA + LoRA+ + DoRA + NEFTune + grad-clip + cosine LR

Inference

Fast, small, long-context.

  • KV cache with optional fp16/bf16 quantisation
  • Prefix caching — save prompt KV state to disk, skip prefill on reload
  • Speculative decoding (greedy) — draft + verify (Leviathan et al., 2023)
  • StreamingLLM sink + window for unbounded context
  • int4 / int8 weight quantisation via MLX, plus HQQ sub-quadratic solver (Badri & Shaji, 2023) and AWQ reader (Lin et al., 2023)
  • LASER selective rank reduction (Sharma et al., 2024)

Interpretability

What is this thing actually doing?

  • Logit lens — per-layer next-token predictions (Nostalgebraist, 2020)
  • Tuned lens — trained per-layer probes (Belrose et al., 2023)
  • Attention heatmap — per-head weights, "watch the model think"
  • Per-layer ablation — zero out attn / mlp / whole block, observe
  • Activation patching — intervene on residual stream at (layer, position) (Meng et al., 2022)

Data

Where the bytes come from.

  • Any HuggingFace dataset via streaming importer (python_ref/fetch_hf_corpus.py)
  • Wikipedia article fetcher (browser-side, in the playground)
  • Project Gutenberg corpus (19 books, ~34 MB — bundled)
  • Magpie synthetic SFT data — bootstrap from a chat-tuned base (Xu et al., 2024)
  • Persistent tokenized corpus cache (skips re-tokenisation across runs)

Run it

In your browser

WebGPU · in-tab training · works on M-series Macs and modern Chrome

The browser playground builds a transformer in WebAssembly + WebGPU, trains it in a Web Worker so the UI never freezes, and lets you watch the loss curve live as the model picks up structure. Every interpretability surface above is wired into a single Sample card — benchmark, logit lens, ablation, all under one collapsible “Inspect & evaluate” panel.

Open the playground →

On your Mac

MLX-Swift · CLI + SwiftUI app · trains models the browser can’t fit

The Mac binary uses MLX-Swift to train preset sizes up to Titan (1.3B parameters) on Apple Silicon’s unified memory. The same architecture ships as a CLI (tinygpt train, sft, dpo, distill, es, laser, hqq, tuned-lens, magpie) and a SwiftUI app for point-and-click fine-tuning.

Build it from source →

Learn how it’s built

Phase-by-phase guides cover every shipped feature with paper references and reproducible commands.

The model from scratch byte-level GPT, attention, MLP, LayerNorm — explained in code Three phases of training pretrain → SFT → DPO, with commands you can paste Knowledge distillation teacher → student with the KL+NLL mix loss Mixture-of-Experts router, experts, load balance, save/load — and the upstream gap that blocks real sparse compute Multi-Token Prediction training-only extra heads — checkpoints stay drop-in compatible Evolution Strategies gradient-free training (Salimans et al., 2017) Interpretability tools logit lens, tuned lens, ablation, activation patching Memory tradeoffs bf16, gradient accumulation, sliding window — what fits in 48 GB Validation report end-to-end runs with actual loss numbers and the bugs the validation caught Phase 9 + 10 status quantisation + architecture menu: what shipped, what’s queued Single-machine roadmap every technique that runs on one Mac, ROI-ranked Leaderboard guide how the three launch benchmarks are scored + how to submit

Numbers

Concrete loss curves and benchmark scores live on the leaderboard. The validation report walks through an end-to-end pipeline that exercises every shipped feature with real corpus + real numbers.