coco · engineering lesson no. 06

I optimized the GPU and it came out 20× slower. The lesson.

This is the most useful entry in the series, because it's about a failure. I had an idea that seemed obviously good, I implemented it carefully, and the result was twenty times worse. Understanding why changed the way I optimize.

Jul 9, 2026·~5 min read GPUnegative resultlesson

The seductive idea: the model's "thinking" part (the FFN) is three operations in a row. Normally each one is a separate GPU launch, and between one launch and the next you have to save intermediate results to memory. What if I fuse them all into one giant launch, with one workgroup per word, keeping the intermediates in the chip's ultra-fast memory? Fewer launches, less traffic. It had to fly.

It flew, all right — straight down. 20× slower.

Why it sank: occupancy

A GPU isn't fast because each core is fast — it's fast because it has thousands of cores working at once. To run flat out it needs a huge number of independent tasks in flight. "One workgroup per word" gives it very few tasks: the GPU is left nearly empty, with most of its cores sitting idle. And the memory traffic I saved is worth nothing if 95% of the machine is stalled.

Each little square is a GPU core. On the left, the normal path keeps almost all of them busy. On the right, the megakernel leaves most of them idle — and a stalled GPU doesn't run, no matter how little it moves.

The lesson, in one sentence

the rule I take away Occupancy beats saved traffic. A fusion that reduces launches but kills parallelism is a net loss. Before fusing, ask: does this leave the GPU full or empty?

The nice thing about the contrast: the fusions that did win in coco are exactly the ones that respect that rule — joining attention with the write to its cache, or collapsing 196 launches into 112, without sacrificing parallelism. The megakernel is still in the code, behind an off switch and with its correctness check intact: a failure kept with affection, in case the rule ever changes.

And speaking of keeping things honestly: the last entry is about how I make the program tell you "not yet" to your face instead of pretending something works. It's number 7.

Previous: multiplying by reading compressed · Index: the notebook