TinyGPT

in-browser playground

Train a small GPT — the kind of model behind ChatGPT, only ~0.8 million parameters instead of a trillion — from scratch, right here in this tab. No server, no install. Watch the loss curve fall, then ask it to write you a sentence. Source ↗

Train

Corpus

Fetch

picks load automatically · or paste your own text below

Model size preset

Hyperparameters — preset above sets these. Click to edit individually.

Model depth layers

Model width d_model

Bytes the model sees at once context

Training steps max_steps

Where it runs backend

Examples per step batch_size

Model —

Estimated run time —

Advanced

How fast it learns learning_rate

Notify me when done

Loss curve wasm

Step 0

Loss –

Time left –

Elapsed –

More metrics

Perplexity–

Holdout error–

Best holdout–

Bytes / sec0

Train ↔ val gap–

Sample

You can sample while training is still running — this is the trick. The model is a single instance in a Web Worker; Generate runs a forward pass against the same weights training is mutating, between steps. Sample twice during a run to watch what the model has picked up.

Starting text prompt

Randomness temperature

How many bytes tokens

Train a model, then generate text here.

What to expect: the model has to be sized to the data. The loss curve falling is the real signal — text quality depends entirely on params × bytes seen.

Your machine detecting… computing…

computing…

Want to go further?

4 Resources curated rabbit holes — 3Blue1Brown, Karpathy, papers, and this repo's own docs

Start here (visual + intuitive)

3Blue1Brown — Neural Networks — the maths, visually
Jay Alammar — The Illustrated Transformer — the diagrams everyone has seen
Karpathy — Let's build GPT — a 2 h build, from zero
Karpathy — Neural Networks: Zero to Hero — the full lecture series

Go deeper (code + papers)

karpathy/nanoGPT — the cleanest small GPT in PyTorch
karpathy/minGPT — the original minimal build
Attention Is All You Need — Vaswani et al., 2017
LoRA: Low-Rank Adaptation — Hu et al., 2021
AdamW — Loshchilov & Hutter, 2017

This project's own write-ups

docs/notes.md — how every layer was built & tested
docs/learn.md — the guided learning path
docs/lora_guide.md — LoRA in detail
docs/performance.md — the SIMD & WebGPU speed work
docs/browser_notes.md — WASM, Workers, OPFS, WebGPU

Related projects

mlc-ai/web-llm — run real LLMs in the browser (inference)
ml-explore/mlx — Apple Silicon ML framework
Hugging Face datasets — where the corpus picker pulls from
MDN — WebGPU — the API powering the GPU path

5 Diagnostics & how to go faster WebGPU matmul benchmark, the Python CLI for 10M+ models, and what makes training fast

Making it faster — what works, in order of impact

Train locally with Python (50–100×). The python_ref/train.py path uses PyTorch with CUDA / Apple MPS and runs on every core. A 10M model trains in ~24 s / 1k steps on an M5 Pro — comfortable iteration speed.
WebGPU backend (≈3–10× on real hardware). The shaders are correct end-to-end (24/24 kernel parity-checked); real-GPU speedup is unmeasured here because the project's CI ran on a software adapter (swiftshader) — see docs/notes.md §10. Try it on your machine and watch tokens/sec.
WASM SIMD (1.6× over scalar WASM). Already on if your browser supports it — the green pill at top confirms. Four floats per cycle in the matmul inner loop instead of one.
Bigger batch (sub-linear). Better cache utilisation per step, fewer kernel dispatches. Memory grows linearly in batch × ctx; the matmul cost dominates so it's near-free up to your RAM limit.
Smaller model (linear). Throughput scales roughly as 1/params for transformer training, because the matmuls grow as params × batch × ctx. Drop d_model from 96 to 48 and per-step time quarters (d² in the inner kernel).
Multi-threaded WASM (4–8×, not implemented here). Would need SharedArrayBuffer and worker threads. Open box.

Full speed write-up: docs/performance.md ↗ · The journey of every lever — shipped, blocked, open, and why: the performance journey ↗

Runs the same matmul on the WebGPU compute kernel and the WASM kernel, checks they agree, and reports the speedup. Needs Chrome / Edge 113+.

Not run yet.

Train larger models locally

In-browser is single-threaded WASM — comfortable up to ~1M params. For 5–25M+ (a few minutes on a laptop), run the Python reference where it uses your GPU (Apple MPS / CUDA).

git clone https://github.com/sarthakagrawal927/tinygpt
cd tinygpt
python -m venv python_ref/.venv && source python_ref/.venv/bin/activate
pip install -r python_ref/requirements.txt

# measure how fast your machine trains:
python python_ref/bench.py

# train a ~10.8M model on your own text:
python python_ref/train.py --model-config configs/model.small.json \
    --data your-text.txt --out checkpoints/run

# generate from it:
python python_ref/sample.py --checkpoint checkpoints/run --prompt "Once "

TinyGPT

Train

Loss curve wasm

Sample

Watch the model think

Top-10 candidates

Attention from last layer · per head

Start here (visual + intuitive)

Go deeper (code + papers)

This project's own write-ups

Related projects

Making it faster — what works, in order of impact

Train larger models locally

Keyboard shortcuts

Welcome to TinyGPT

Train ?

Loss curve ? wasm

Sample ? live

Watch the model think

Top-10 candidates

Attention from last layer · per head

Start here (visual + intuitive)

Go deeper (code + papers)

This project's own write-ups

Related projects

Making it faster — what works, in order of impact

Train larger models locally

Train

Loss curve wasm

Sample