coco · performance no. 04

Why a 3 GB model runs faster than a 14 GB one (and when it stops)

It sounds odd: you take a model, compress it to a quarter of its size, and suddenly it generates faster. It's not magic — it's that generating text isn't limited by what the computer computes, but by how much memory it has to move. And there's an honest catch here that I'm going to show you.

9 Jul 2026·~6 min read performanceGPUmemory

To generate a single word, the model has to read all its weights — the millions of numbers it learned. On a modern GPU, reading those numbers from memory takes longer than multiplying them. Put another way: the bottleneck is memory, not the calculator. The calculator is almost always waiting for the data to arrive.

If that's true, the consequence is direct: a model that's 1/4 the size is read 4 times faster, and therefore generates faster. That's why my model compressed with turboquant beats an uncompressed one even though it does exactly the same arithmetic. The headline:

measured · short context A 7B model compressed to 3.2 GB generates 1.55× faster than the same model uncompressed at 14 GB — reading fewer bytes per word. Same computation, four times less traffic.

The catch: when the context grows, the thesis flips

Here's the honest part. The weights aren't the only thing you have to read. Every word the model has already seen leaves a "note" in memory (they call it the KV cache), and to generate the next word you have to reread all of those notes. With little context, those notes are negligible next to the weights → the weights rule → compressing them wins. But with a lot of context, the notes grow and grow until they dominate the traffic — and at that point compressing the weights barely matters, because the problem has become something else.

With little context, reading the weights is almost all the work → compressing them speeds things up. With a lot of context, rereading the KV cache eats up the traffic → weight compression stops being the lever.

Move it yourself: find the point where it crosses over

Slide the context and watch how coco's advantage shifts against a reference engine (relative numbers, 1.5B model). Top-left it wins; slide it to the right and you'll see where it crosses 1.0× and turns into a disadvantage:

That crossover isn't a flaw to hide: it's where the technique lives. Compression is a lever for latency at short and medium context, not a universal accelerator. Selling it as "always faster" would be lying; telling you where it stops working is what lets you trust the rest.

What's left to do

The reason it loses at long context has a name and a fix: the "read the whole context" phase is still quadratic (it grows with the square of the length). Block-wise processing would flatten it — and that's exactly where the GPU kernels in the next post are aimed.

Previous: shrinking a model 3× without data · Index: the notebook