coco · inference engine no. 01

Every new AI model breaks the tooling. I built an engine that shrugs it off.

Every time a new architecture ships, the programs that run AI models have to be rewritten almost from scratch. I got tired of that and built an engine with a single swappable part. Let me show you what's inside.

8 Jul 2026·~7 min read RustGPUarchitecture

An AI model is, on the inside, a tower of identical layers stacked on top of each other. Each layer does two things: it mixes information between the words (the famous part, "attention") and then thinks about each word on its own. The whole tower repeats dozens of times, and out comes the answer.

The practical problem is this: every lab ships its model with a slightly different way of doing that mixing. One uses classic attention. Another compresses it to save memory. This month's model invents a "linear attention" that never re-reads the past. And the programs that run these models —the inference engines— usually have one huge, separate file per architecture. A new model comes out, and it's copy-paste-and-pray.

Left: the usual pattern — one giant file per architecture. Right: coco reduces each model to one row in a registry and a single swappable part inside a generic decoder.

The idea: a single seam

The key observation is that modern architectures differ only in how they mix information between words. Everything else —normalizing, the "thinking" part, stacking layers— is identical. So in coco all of that is generic code written once, and the only part that changes is a plug-shaped slot: the SequenceMixer.

A SequenceMixer is a contract: "give me the words, give me back the mixed words." Any form of attention that honors that contract fits into the slot. Today I have three parts that fit:

Three parts fit into the same slot today: classic attention (Qwen), DeepSeek's compressed attention (MLA), and a recurrent linear attention (GatedDeltaNet). The rest of the tower never notices.

Watch it run

Here's the engine generating a response. On the right, the telemetry: every token that comes out passes through the model's 24 layers, and each layer consults its mixer. Notice how the full attention layers have to re-read the entire context (the "re-reads memory" bar rises), while the linear attention ones keep a fixed state and barely spend anything. Hit play:

↑ A design illustration with a 3:1 hybrid tower (Qwen3.5 style). Today Qwen (all full attention) and DeepSeek (all MLA) actually run; the GatedDeltaNet part is validated and landing.

what works today Four full architectures through the same decoder — Qwen2, Qwen3, Qwen3-MoE and DeepSeek V2/V3 — with three swappable mixers. It compiles clean and passes 12/12 of its tests. Adding the fifth isn't rewriting the engine: it's filling the slot.

Why I care

Because the pace of new architectures isn't going to slow down — it's going to speed up. An engine that treats every model as a special case spends its life chasing the latest paper. An engine with a single seam turns "supporting the model of the moment" into an afternoon's work, not a month's.

And there's an uncomfortable honesty in all this that I like to show: the fifth architecture (a very recent hybrid) coco recognizes but still refuses to run, with an error message that tells you exactly what's left to plug in. I'd rather have a loud, honest "not yet" than a fake green path. But that's another entry.

Next in the notebook: how I translated a new "attention" into Rust without missing a single decimal — the discipline of validating against the truth before wiring anything up.