coco · validation no. 02

How I ported a new kind of "attention" to Rust without missing a single decimal

When you rewrite a model's math in another language, the worst bug isn't the one that blows up: it's the one that almost works. The model responds, it looks reasonable… and it's subtly broken. Here's how I make sure that doesn't happen.

9 Jul 2026·~6 min read Rustvalidationmethod

In the previous entry I explained that my engine has a slot where different ways of "mixing" information between words plug in. This month it was time to wire in a new and peculiar piece: GatedDeltaNet, a "linear" attention that, instead of rereading the whole past, keeps a state it updates as it goes — like someone carrying a mental summary instead of rereading the entire book.

The piece is a handful of chained math operations. And here's the trap: if you get a sign wrong, mix up the order of two multiplications, or miss a normalization detail, the program doesn't complain. It compiles, it runs, and the model says things... just slightly worse ones, without you knowing why. A silent bug in the hardest place to look.

The rule I follow: I never wire in a new piece "to see if it works." First I validate it in isolation against a source of truth, with a hard numerical bar. If it doesn't clear it, it doesn't go in.

The idea: manufacture a "truth" and chase it

The trick is simple and surprisingly powerful. I take the reference implementation (the authors', in Python) and feed it some arbitrary input numbers. I save its exact output to a file: that's my oracle, my lab-grade truth. Then I reimplement the piece in Rust and feed it exactly the same numbers. Its output has to match the oracle as far as the computer's precision reaches.

The same input runs through the reference (which we freeze as the oracle), through a quick numpy check, and through the Rust reimplementation. Both have to match the oracle below a brutally strict bar: a maximum difference of 1e-4 (0.0001).

See for yourself

Here are the first real values my Rust version spits out, side by side with the oracle's "truth." Look at the third column: the difference. Hit compare and watch how many decimals they agree on — and where the maximum error lands over the whole result:

result Maximum difference 1.825 × 10⁻⁷ (mean 3.5 × 10⁻⁸) against a bar of 10⁻⁴. In other words: it matches the reference down to the seventh decimal place. That's no longer "it seems to work": it's the same function, rewritten in another language.

That tiny remaining difference isn't a bug: it's the inevitable noise of two computers adding the same numbers in a slightly different order. That's why the bar is 1e-4 and not zero — you have to leave room for arithmetic, but slam the door on real errors, which would be a thousand times bigger.

The uncomfortable honesty

And now the part I most like to tell: passing the oracle doesn't mean the model works. It means this piece is correct. In fact, as I write this, the compiler throws me a warning: "nobody uses this piece." It's true — it's validated and stored away, waiting for me to finish wiring in the rest of the architecture around it.

I could have wired it in already and pretended everything works. I prefer an honest in-between state: correct in isolation, not yet connected. How I get the program to tell you that to your face instead of faking it is another entry.

Previous: every new AI model breaks the tools · Index: the notebook