Bitácora · Kiko Cisneros
coco · GPU kernels no. 05

The GPU trick: multiplying straight from compressed data, without unpacking it

Compressing a model is only worth it if you don't decompress it to use it. Sounds obvious, but that's exactly where almost everyone loses the advantage. The trick is to reconstruct each number inside the GPU, mid-multiplication, without ever writing it out in full to memory.

Jul 9, 2026·~5 min read MetalGPUkernels

In the turboquant post we saw that weights are stored compressed: each number is a small code that points into a dictionary, plus one scale per block. Perfect for disk. The problem is multiplying: the model's star operation expects full numbers, not codes.

The naive way would be: decompress everything to memory (rebuild the whole tensor in 16-bit) and then multiply. But that undoes the entire advantage — you end up moving exactly the same bytes you were trying to save. It's like packing your suitcase tight and then pulling everything back out before boarding the plane.

the naive path codes expand to 16-bit(writes to memory) multiply ✗ moves all the bytes again the coco path codes multiply while reconstructingin the register (no memory) ✓ the weight is never written out in full
The naive path rebuilds the full tensor in memory and loses the advantage. coco reads the code and turns it into a value right inside the register, just as it's about to multiply — and that's it. The full weight never exists in memory.

Watch it decode, code by code

Here's the heart of the kernel. Compressed codes come in; each one is looked up in the dictionary, multiplied by the block's scale and by the input, and accumulated — a dot product. Notice that a full weight tensor never appears: each value lives for an instant in the register and then vanishes. Hit play:

also Fusing operations that used to be separate GPU launches cuts the coordination work from 196 to 112 launches per token. Less back-and-forth, more time actually multiplying.

The honest part

This is specific to Apple's GPU (Metal) and it isn't free to write: there are kernel variants that split the work in different ways, and you have to pick the one that fills the GPU best. In fact, my most aggressive attempt to fuse everything came out 20 times slower — and that story, which is the most useful of all, is the next post.

Previous: the bandwidth thesis · Index: the notebook