TinyGPT — train a tiny GPT in your browser

Train a byte-level GPT from scratch — entirely in your browser, in a Web Worker, on WebAssembly. The UI thread stays responsive throughout.

Your machine — and what it can train

detecting…

1 · Training corpus

context length layers d_model

max steps learning rate batch size backend

2 · Loss

step 0 train – val – 0 tok/s ETA – backend wasm

3 · Sample from the model

temperature tokens

Train a model, then generate text from it here.

4 · WebGPU matmul benchmark

Runs the same matrix multiply on the WebGPU compute kernel and on the WASM kernel, checks they agree, and reports the speed-up. WebGPU-capable browser required (Chrome / Edge 113+).

Not run yet.

5 · Train larger models locally

In-browser training is single-threaded WebAssembly — comfortable up to roughly 1M parameters. To train bigger (5–25M+ is fine on a laptop), run the Python reference locally, where it uses your GPU (Apple MPS / CUDA). bench.py measures what your machine can handle; configs/model.small.json is a ready ~10.8M model.

git clone https://github.com/sarthakagrawal927/tinygpt
cd tinygpt
python -m venv python_ref/.venv && source python_ref/.venv/bin/activate
pip install -r python_ref/requirements.txt

# measure how fast your machine trains, at several model sizes:
python python_ref/bench.py

# train a ~10.8M model on your own text:
python python_ref/train.py --model-config configs/model.small.json \
    --data your-text.txt --out checkpoints/run

# generate from it:
python python_ref/sample.py --checkpoint checkpoints/run --prompt "Once "