Bitácora · Kiko Cisneros
coco · compression no. 03

Shrinking an AI model 3× with no data and no quality loss

A good model doesn't fit on an ordinary laptop. The usual way to shrink it needs a pile of calibration data and still loses some quality. I wrote a compressor that needs no data, fits in a single file, and in the cases I checked gives exactly the same answer.

9 Jul 2026·~6 min read RustquantizationGPU

A model's weights are, literally, millions of decimal numbers. Storing them "as is" takes up a huge amount: each number uses 16 bits. The idea behind quantizing is to store each number with fewer bits — say 3 instead of 16 — the way you'd round it off. The tricky part is doing it without making the model dumb.

My compressor is called turboquant, and here's the headline:

measured · Qwen2.5-1.5B From 2944 MB to 934 MB — the whole model fits in under 1 GB (3.15×). The big matrices alone shrink 5.12×. And the output is byte-for-byte identical to the uncompressed model on the examples I verified (same facts, same fluent paragraph).

How so much fits without breaking

The trick has three steps, and none of them needs data. First the block of numbers is "rotated" with a reversible transform that spreads the information more evenly — so no weird, huge number spoils the rounding of its neighbors. Then the values are grouped into a small dictionary of codes (a handful of representative values), and each weight is stored as "code number such-and-such." Finally, a scale is recorded per block to recover the magnitude. Reconstructing is just undoing the rotation.

1 · rotate spreads the values 2 · dictionary each weight → a code 3 · scale 1 number per block lives on the GPU compressed unpacked on the fly
The three steps, without a single piece of calibration data. And the part that matters for speed: the weights stay compressed inside the GPU and are decompressed at the moment they're used — never re-expanded to their original size.

Watch it read compressed

Here the compressed model answers. In the telemetry, watch how many bytes it reads per word: because it's compressed, it moves far less memory. That detail — which looks like plumbing — is exactly what makes it fast, and it's worth another entry.

What isn't perfect yet

Two honest caveats. One: the real 3-bit packing (the 934 MB one) is tested and works, but the default binary still comes out in a slightly larger 4-bit format — the 3-bit one is a button-press away, not fully wired in yet. Two: I've verified "same answer" through coherence (same facts, same text), not with a full precision benchmark. I'm not going to sell you that nothing is ever lost; I'm telling you that in what I looked at, you can't tell the difference.

Previous: validating without missing a single decimal · Index: the notebook