Two runtimes, four gigabytes, one browser tab.

Two runtimes, because one is not enough

There is no single way to run a language model in a browser today, so the app carries two and picks at boot:

There's also a third mode with no download at all — a rule-based brain — so the very first interaction works while a 4 GB file is still arriving. A blank screen with a progress bar is how you lose someone in the first ten seconds.

Bug #1: navigator.gpu exists, and lies

The obvious WebGPU check is the wrong one. On several Linux/driver combinations, sandboxed environments and headless browsers, navigator.gpu is present as an API object with no usable adapter behind it. Our first version trusted it, so the sequence was:

Download the entire model, then fail on "Failed to get GPU adapter", then — because a failure at that layer escalated to the bigger brain — start downloading Gemma too. Several gigabytes, for nothing, on the exact machines least able to afford it. The fix is one await:

let device = 'wasm';
if (navigator.gpu) {
  try { device = (await navigator.gpu.requestAdapter()) ? 'webgpu' : 'wasm'; }
  catch { device = 'wasm'; }
}

Bug #2: the quantization that produces beautiful garbage

transformers.js lets you pick a dtype for the ONNX weights. q4f16 is smaller and faster than q4, and on WebGPU — even with shader-f16 available — it produced fluent, confident, completely meaningless text. Not a crash. Not a warning. Just wrong tokens, indistinguishable from a bad model.

That's the worst class of bug in this whole area: silent quality loss. It doesn't show up in a smoke test that checks "did we get output." It shows up as "this model is dumb," and you go looking in the wrong place for a week. The code now carries the scar as a comment:

export const MODEL = {
  id: 'KikoCis/Elffuss-LM-1.2B-ONNX',
  dtype: 'q4',   // NOT q4f16! it generates garbage via WebGPU (verified)
};

Bug #3: the service worker that broke the thing it was meant to accelerate

Caching gigabytes sounds like exactly what a service worker is for, so we wrote one: intercept the requests to the weights, keep them in Cache Storage, serve them from there next time. It worked for the LiteRT bundle and it broke ONNX loading completely.

The reason is a detail of how large ONNX models are packaged: the weights live in a separate external-data file (model_q4.onnx_data) that the runtime re-fetches through its own machinery. Intercepting and replaying that request produced ERR_FAILED, and the model simply never loaded.

The interesting part is what we found while debugging: the service worker was unnecessary in the first place. transformers.js already caches into Cache Storage on its own, and our LiteRT loader does its own cache-first fetch. So the fix wasn't a smarter service worker. It was deleting the idea:

It's still registered — as an inert no-op — for one boring but important reason: to cleanly replace the previous version in browsers that already installed the broken one. A service worker you can't reach is a bug you can't fix remotely.

The part that actually needed writing: cache-first, by hand

For the Gemma bundle we do fetch and cache ourselves, because LiteRT's internal loader pulls the weights with XHR+Range from inside a worker, which sidesteps the cache we want. So the loader downloads the .litertlm, tees the stream — one half drives the progress bar, the other half goes into Cache Storage — and hands the runtime a Blob:

const cache = await caches.open(MODEL_CACHE);
const hit = await cache.match(url);
if (hit) return await hit.blob();          // second visit: no network at all
const net = await fetch(url);
const [prog, toCache] = net.body.tee();     // progress ← → cache, one download
await cache.put(url, new Response(toCache, { headers }));

And before any of that, the app asks for navigator.storage.persist(). Without that grant, a browser under storage pressure is entitled to evict your 4 GB — and the user, who did nothing wrong, waits for a full re-download. With it, the second visit starts in about a second.

How much you actually download

The context ladder

How much context you get isn't a constant — it depends on the bundle and on how much GPU memory the machine will hand over. Rather than hardcode a safe-but-small number, the loader negotiates downward until something works:

const CTX_LADDER = [32768, 16384, 8192, 4096];

A gaming desktop ends up with 32k. A thin laptop with an integrated GPU may land on 8k, and it still works — because the harness above it assumes context is scarce and manages it aggressively. That's the next post.

The honest edges

• First load is genuinely expensive. 4 GB on a first visit is a real cost that a hosted API doesn't have. We hide it behind an instant no-download mode, but we can't remove it.
• We do not ship our own weights as the brain. We healed a Gemma-4 E4B ourselves, and it doesn't load in the browser: the runtime wants the artisan packaging and our export is prefill_decode. So the shipped brain is Google's base build, and the agentic behaviour lives in the system prompt instead. Honest version: the model is theirs, the agent is ours.
• WebGPU is still young. Long inference dispatches can trip the GPU watchdog; driver differences are real; and there is no way to test "every GPU" from one laptop.

Runtime layer source: js/providers/litert.js, js/providers/onnx.js, js/model-cache.js, sw.js in elffuss-claw / elffuss-code. Next: the harness.