The performance journey

The C++ matmul is compiled twice — once scalar, once with -msimd128. With SIMD on, four f32 lanes multiply per cycle in the inner loop instead of one. docs/performance.md reports ~1.6×; current build is SIMD by default (the numbers in the Benchmark log above are SIMD-on).

The page's "WASM SIMD" pill at top shows whether your browser actually loaded the SIMD build. All Chromium-family browsers and Safari 16.4+ do.

The full training loop runs on the GPU — all 24 kernels written in WGSL, every one finite-difference and parity-checked against the WASM reference. Correct end to end.

Measured curve: Small 2.6×, Medium 6.8×, Large 9.3×, XL 12.1× over the multi-threaded WASM SIMD baseline, via tests/test_webgpu_train.mjs. Loss drift 1.1%–2.5% across the curve — float-reorder noise. The earlier single-preset "~7× on Small" number predated the multi-threaded WASM baseline and is withdrawn.

matmul_forward and matmul_backward now split the M dimension across CPU cores via std::thread. Each thread takes a contiguous row slice; outputs don't overlap so no locks. The dB path is the exception — it accumulates over M, so we use per-thread scratch and a final reduction. Threading only kicks in when M ≥ 64.

The pthread WASM build requires SharedArrayBuffer, which requires cross-origin isolation. The _headers file sets COOP/COEP for Cloudflare Pages; vite.config.ts mirrors it for the dev server.

Why only 2×, not 4-8×: the workload is memory-bandwidth bound past ~2 threads. Each matmul reads the entire B matrix; that's the shared bottleneck. Adding cores past the BW limit gives diminishing returns. Real measurement consistent with this theory.

Tiled matmul (Tm=32, Tn=64, Tk=32 blocks) was implemented and benchmarked against the baseline on the same single-threaded WASM-SIMD build. The result:

Why the theoretical prediction (1.5-2×) didn't materialise here: the baseline matmul's inner loop is a fixed-bound for n in 0..N that emcc -O3 -msimd128 aggressively autovectorises into f32x4 FMA chains. The tiled variant introduces variable-bound inner loops (for n in n0..n1) that the autovectoriser handles less cleanly, so the SIMD win shrinks just as the cache win arrives. Net: wash.

The in-browser experiment with fp16-packed weights on top of tiled matmul (lever 10b) showed no compound win — once tiling has amortised global-memory traffic, halving bandwidth has nowhere left to help. A full mixed-precision refactor (loss scaling, fp16 gradient accumulators, fp16 variants of every kernel) would be multi-week work for a marginal return on the sizes we run in browser.

Where this belongs instead: the native macOS app (lever 19). MLX-Swift supports bf16 natively, with hardware accelerators on Apple Silicon. The training loop's the same algorithm; the host's the right place to spend the complexity.

Standard attention materialises an N×N score matrix in memory; Flash Attention computes it in tiles so the full matrix never exists — saving memory and beating naïve attention on speed by avoiding HBM round-trips.

What changed: with the new Huge/Massive/Mega presets, ctx now goes to 256–512 — the regime where attention's share of step time goes from ~12% (ctx 64) to ~40% (ctx 256) to ~55% (ctx 512). At Mega (ctx 512), the score matrix at B=2, H=12, fp32 is ~25 MB per attention call — starting to hit WebGPU buffer pressure.

Estimated impact, today: ~1.18× on Massive, ~1.7× on Mega. The ctx=512 preset is the first where Flash Attention becomes the highest-ROI open lever.

Fusing forward + loss + backward + AdamW into one mega-kernel would kill some memory traffic — but the project's stated principle is "every layer can be understood." Fused kernels are notoriously unreadable. The right move would be a second build target ("fast" alongside "readable") which doubles maintenance for a modest gain.

On bigger models, matmul dominates anyway and dispatch overhead is in the noise. With blocked4 + FA2 shipped, the speedup curve already climbs from 2.6× (Small) to 12.1× (XL) — the structural wins are spent. Some fusion does exist where readability wasn't a sacrifice: AdamW + grad clip ride in one pass, residual + layernorm chain without intermediate writeback.

The Python reference runs the same model on PyTorch with full GPU support. On an M5 Pro: 10M-param model trains at ~24 s per 1,000 steps. Practical iteration speed for real models.

This is the right answer for anyone serious — the in-browser path is for learning the mechanics, not training the next ChatGPT. The Diagnostics section of the playground has the three commands you need.

V8 caps each tab's WASM heap near 4 GB on 32-bit pointers — that's ~250M fp32 params with Adam state, full stop. Memory64 (-sMEMORY64=1 -sWASM_BIGINT) switches the module to 64-bit pointers and lifts the cap into the tens of GB on Chromium 133+.

Measured (when called directly): 473M-param model (~5.6 GB heap with Adam state) allocates cleanly in Node — a config that hard-OOMed on the 32-bit module.

Caveat (task #66): the 64-bit module's JS↔WASM ABI wasn't being exercised by the existing tests/bench_wasm.mjs (which loads the 32-bit module), so a cwrap pointer-conversion bug shipped — _malloc returns Number but pointer args expect BigInt. The browser was calling into this broken bridge for XL and bigger; the loader now falls back to the 32-bit module for XL/Massive/Mega/Behemoth in-browser. Reproducer: tests/test_wasm64_xl_node.mjs. Lesson captured in docs/lessons.md.

Stacks two well-known wins. (a) Same 16×16 workgroup-shared tiling as lever 10, plus (b) each of the 256 threads computes a 4×4 block of output values held in registers. Workgroup outputs a 64×64 tile; each shared-memory load gets reused 4× across the thread's register accumulator via outer-product structure. Arithmetic intensity per shared-mem load climbs from ~1 fused multiply-add to ~16 — well past the point where matmul becomes compute-bound rather than memory-bound.

Measured on M-series WebGPU:

Speedup grows with matrix size because bigger problems amortize workgroup-shared-memory loading more effectively across the 4×4 register reuse. At 2048³ (the kind of shape that shows up in Mega and Behemoth presets) the kernel runs 5.18× faster than the naive version and 1.89× faster than the merely-tiled one.

Tried scaling the register block up from 4×4 to 8×8 (workgroup output tile 128×128 instead of 64×64). Hypothesis was that 4× the arithmetic intensity per shared-memory load would translate to ~1.5× more speedup on top of blocked4. Lost at every size:

Most likely cause: 64 floats per thread for the accumulator exceeds the per-thread register budget on Apple GPUs, forcing register spill into local memory and tanking effective compute throughput. Lower workgroup occupancy (16 KB shared per workgroup vs 4 KB) compounds it — fewer concurrent workgroups per SM. Kept in the codebase as a documented negative result. Same lesson as f16-vs-tiled: more aggressive is not always faster; benchmark every variant.

Classic 16×16 tiled matmul using var<workgroup> shared memory (textbook Goto/VandeGeijn pattern). Each workgroup of 16×16 threads cooperatively loads A's and B's 16×16 blocks into shared memory, then each thread does 16 multiply-accumulates from shared. Effectively turns 16 global reads into 1 global + 16 shared, which on big matmuls is where the GPU starts looking like a GPU.

Measured on M-series WebGPU, dispatch-only timing:

Clean ~2.5× across every realistic size, peak ~2.7× at 512³. Parity validated at sizes ≤ 512.

Weights live as packed half-precision (two f16 per u32 via pack2x16float built-ins), accumulation in f32. The standalone version beats naive by ~1.7× by halving global bandwidth. But once tiling is in place the kernel is no longer bandwidth-bound — it's compute-bound on shared-memory ops — and halving global bandwidth no longer helps. The combined tiled+f16 kernel is the same speed as plain tiled at 1024³ and a touch slower at 2048³ (17.78 ms vs 16.90 ms).

Lesson: always bench an optimization against the best baseline, not the naive one. The ~1.7× we measured earlier was real but not additive — it was a different way to get the same memory-traffic win that tiling already captures more thoroughly.

The packed kernel stays in the codebase as a reference + for cases where the model genuinely needs more total bytes than the GPU can hold (Behemoth-scale weight buffers), where halving storage isn't just about speed but about fitting at all.

Subgroup intrinsics (subgroupAdd, subgroupMax) would shave a tree-reduction's log₂(blockDim) passes into one on softmax/layernorm/attention reductions. The shipped kernels already use workgroup-shared reductions, which on Apple GPUs are within 10–20% of what subgroups would deliver — and reductions are 5–8% of total step time on every preset we ship. The lift on the bottom line is ~1–2%; not worth the added kernel surface area, the "subgroups" extension gate, or the test plumbing.

Workgroup-cooperative forward — one workgroup per (batch, head, Q-tile of 16 rows), K and V walked in blocks of 16, online softmax in registers across K blocks. Default attention path when hd ≤ 64 (every preset up to Behemoth). Lives in webgpu/attention_fa2.wgsl.

Backward kernels (attn_dscores_fa2 + attn_dv_fa2) reconstruct P = exp(S − L) from q/k and the saved log-sum-exp instead of reading the cached attn matrix. That removed the forward's second-pass writeback entirely; on Mega-class shapes (B=4, H=8, T=512) ~67 MB of global memory traffic per layer per step now stays on-chip.

Folded into the macOS app (lever 19). Native MLX-Swift supports adapter training as a first-class operation; bringing the same feature to the WebGPU path would require a parallel set of WGSL kernels (LoRA-aware matmul, restricted optimizer step) for a duplicate-of-Python win. The Python reference at python_ref/lora.py stays as the canonical implementation.

Folded into the macOS app (lever 19), where the quantization libraries are mature (MLX, llama.cpp-style GGUF). Int4/int8 in WGSL is doable but the test surface is large — every kernel needs a quantized variant plus parity tests — and the immediate need isn't sharp: Behemoth-class models already fit via Memory64 in browser. The win-to-effort ratio is poor relative to spending the same weeks on the native path.

A drop-in Newton-Schulz orthogonalisation of matrix-shaped gradients before stepping. Empirically matches AdamW in fewer steps. Skipped here because faster-convergence isn't the project's constraint — readability and per-step compute are. Listed in docs/feature_ideas.md for whenever someone wants to port it as a contributor experiment.

Newer SIMD ops (FMA, dot products, relaxed rounding) the older -msimd128 set leaves on the table. Both the 32-bit and 64-bit WASM builds enable -mrelaxed-simd; compilers emit the new opcodes where they help and fall back to the baseline SIMD otherwise. Runtime requirements: Chrome 114+, Firefox 120+, Safari 18.4+ — anywhere our COOP/COEP'd pthread build already runs.

Verified by re-running tests/test_wasm64_xl_node.mjs after the rebuild: same losses (5.57 → 3.04 over 5 XL steps), within run-to-run variance on per-step time. Free correctness; modest opportunistic speedup on the kernels the compiler decides to vectorise more aggressively.

A Load from gallery button on the Setup screen opens a dialog with 4 cards: same architecture (12L, d=256, ctx=256, ~9.6M params), four different corpora. Each shows a sample, params, training loss, and one click loads the .tinygpt file through the same path as model upload.

All four models were trained in this browser via browser/train_gallery_one.mjs (Playwright drives a real Chromium tab against the dev server, 5000 steps, WebGPU). v1 is bundled in browser/public/gallery/; manifest.json drives the dialog so future entries drop in without code changes. v2 will move to R2 (≈ $0.015/GB-month, zero egress to Pages) once we cross the bundled-asset budget.

Pairs naturally with lever 18 (diverse data structures) — each gallery entry demonstrates a different kind of pattern the same architecture can pick up.

The playground accepts any UTF-8 text today, but the bundled demo and most curated corpora are English prose. What the architecture can actually learn isn't just language — it's any pattern with local + long-range structure. Worth demonstrating with a handful of materially different source types:

• Tabular data (CSV / Excel-exported sheets). The model learns the row/column rhythm — commas and newlines in the right places, repeated header tokens, value-range patterns per column.
• Songs (lyrics + chord sheets). Verse/chorus repetition, line length conventions, the way a chord line sits above a lyric line.
• Full books. Chapter structure, dialogue attribution, prose vs. dialogue cadence. Already partway there with the Shakespeare demo, but a long-form prose book exposes the model's handling of paragraph- and chapter-scale structure.
• Code. Indentation, balanced brackets, function signatures, the convention that def is followed by a colon and an indented block. A test of the model's hierarchical reasoning.

Each one is a 30-min-to-few-hours training run. The interesting move isn't the data plumbing (which already works — just paste UTF-8 or pick a Hugging Face dataset); it's the presentation: each gallery card shows side-by-side input format vs. generated output, so the visitor sees that the same architecture picked up the structure of whatever it was fed.

Three feature buckets that genuinely belong together — each one multiplies the value of the other two — and that all benefit from the same native ML primitives:

• bf16 / fp16 weights and activations. Halves the memory footprint at training and inference time. First-class on Apple Silicon, with hardware accelerators. The in-browser fp16-packed experiment showed no compound gain (lever 10b); native is where it actually pays.
• int8 / int4 quantized inference. Load a quantized checkpoint, sample without dequantising the full weight matrix. Lets the 9.6M-param Huge model ship at < 10 MB and lets much bigger models run sample-only without exceeding heap. MLX + GGUF-style formats handle this off-the-shelf.
• LoRA fine-tuning. Load a base checkpoint, freeze the weights, train low-rank adapter matrices on your own corpus. The Python reference already supports this (docs/lora_guide.md); the hook is "load a gallery model → click Fine-tune → watch the voice shift in a few minutes." Pairs directly with lever 17.

Doing these in the WebGPU path was previously listed as three separate planned items. The honest read: each one would mean a parallel set of WGSL kernels (quantized matmul, LoRA-aware matmul, fp16 variants of every op), parity tests against the fp32 reference, and a doubled test matrix. Months of work for capability that the native ML framework gives in days. Better to ship them together on the native side first; back-port to in-browser only if real usage demands it.

A native macOS app — SwiftUI shell, MLX-Swift training loop. Same model architecture, same .tinygpt file format. Train on your Mac, drop the checkpoint into the browser playground anywhere; load a browser-trained model into the Mac app to continue training at much higher throughput.

How much faster — and why. Same Huge model (12L, d=256, ctx=256, 5000 steps) on the same M-series Mac:

• WebGPU (today): ~60 min · baseline
• MLX (Python): ~5–6 min · ~10×
• MLX-Swift + hand-tuned Metal: ~2–3 min · ~20–25×
• + Apple Neural Engine for inference: sampling ~30–50× faster

The compounding wins: unified memory (no upload/download tax — WebGPU pays this on every buffer round-trip); Metal Performance Shaders' GEMM kernels tuned per Apple GPU generation; ~3× the effective memory bandwidth (M3 Max: 400 GB/s vs WebGPU's ~150 GB/s effective); ANE (16-core, ~38 TOPS on M3) which WebGPU cannot reach; async compute queues overlapping compute + memcpy.

The architecture also lifts the parameter ceiling. Browser caps out around ~10M params before user experience falls apart (training-time budget, memory, GPU sharing with the compositor). MLX-Swift can comfortably train 100M–1B+ models on the same laptop — same code path, just a faster runtime.

The boundary between what belongs in-browser and what belongs native is in docs/shared_vs_native.md. Together with the gallery (lever 17), the diverse-data milestone (lever 18), and the quant + LoRA bundle (lever 19, which lands here), this is the remaining work.

Storage-side 4-bit quantization for the gallery models. Block-wise symmetric scheme (block size 64, one fp16 scale per block, two int4 indices packed per byte — same shape as GGUF Q4_0). Conversion runs offline via finalize_gallery_int4.mjs against the existing fp16 .bin files; the browser dequantizes block-wise back to fp32 once at load time and hands the canonical layout to the existing WASM importer, which doesn't know anything changed. The fp16 files keep shipping alongside as the fallback.

Why not int4 GEMM on the GPU? Apple's WebGPU implementation lacks the integer matmul intrinsics that would make on-device int4 compute a win — the actual int4 multiply would still decode each weight before fma. Trying to do it in WGSL ends up slower than dequantizing once at load time and running the existing fp16-storage matmul. The honest goal here is download size + cold-load time, not inference speed: cold-start drops by 75% on the gallery path.

Quality. Gated by two checks: (1) a startup numerics gate (runInt4NumericsGate) that synthesizes a representative weight matrix and verifies the round-trip error fits the inherent 4-bit envelope — catches catastrophic bugs (wrong block size, endianness) without enforcing an unrealistic per-element bound; (2) the browser/smoke_int4.mjs node-side check that the published files round-trip with bounded per-tensor drift. End- to-end: the Shakespeare model still generates Shakespeare (see smoke_int4_browser.mjs).

A few in-browser speedups exist on the frontier but aren't ready for production users in 2026. Logged here so future-us knows exactly where to look when the spec or implementation lands.

WebGPU cooperative matrix (wmma / tensor-core mapping)

• What: a WGSL extension exposing matrix-multiply-accumulate hardware (NVIDIA tensor cores, AMD MFMA, Apple AMX).
• Expected gain: ~3–5× on NVIDIA / AMD; ~1.3× on Apple (Apple's AMX is less directly reachable via this path).
• Status (May 2026): behind chrome://flags/#enable-unsafe-webgpu + --enable-features=WebGPUExperimentalFeatures. API still moving.
• Skip rationale: Apple gain is the small case for our M-series target. NVIDIA gain doesn't help most browser users. Won't ship until API stabilises and Apple win improves.

WebNN — route to OS NN runtimes (CoreML / DirectML)

• What: a Web API that hands neural-network graphs to the operating system, which then runs them on CoreML (Apple) / DirectML (Windows) / TFLite (Android). Can route to the Apple Neural Engine.
• Expected gain: 3–5× on Apple / Windows for inference; ~2× for training where supported.
• Status (May 2026): Chrome 126+ behind chrome://flags/#enable-webnn-api. Training support is minimal; mostly inference. API changing monthly.
• Skip rationale: training-side coverage is still thin. Revisit when CoreML/DirectML backends support backward passes and the API freezes.

Async compute queues — overlap compute and memcpy

• What: WebGPU lets you create multiple queues; submit copy commands on one while compute runs on another. Hides upload/download latency.
• Expected gain: ~1.2–1.5×, mostly during sampling and the small handful of CPU-touching steps.
• Status: supported today.
• Skip rationale: our hot path is GPU-bound, not transfer-bound — once the first batch is uploaded, training never crosses the boundary. Worth picking up if we ever stream training data dynamically (e.g., gallery-scale corpora that don't fit in tab memory).

Hand-tuned WGSL kernels per matmul shape

• What: the current matmul kernel is one general-purpose blocked variant. Writing per-shape kernels (e.g., one specifically for the 4 × dModel × ctx attention output, one for the dMlp × dModel MLP), each with workgroup sizes tuned to that shape, would close the gap to the theoretical ceiling.
• Expected gain: ~1.5–2× on the matmul path.
• Status: pure engineering — no spec dependency.
• Skip rationale: weeks of work for a modest gain when the Mac app gets us 20–30× by switching runtimes entirely. We'll only come back here if Apple Neural Engine access to the web never materialises and we're still on WebGPU long-term.

If we ever stacked all of these

Stacked gains compound, but with diminishing returns on top of today's shipped stack: coop-matrix (1.3× on Apple) × per-shape kernels (1.7×) × async queues (1.2×) ≈ ~2.6× over today. Still an order of magnitude short of the Mac app's ~30×. That's the structural fact behind lever 20.