My 27B kept losing to a 4B. Most of that was my fault.
My browser stack can run three models: Gemma 4 E4B, a 2-billion-parameter MiniCPM, and a 27-billion-parameter ternary model, Ternary Bonsai 27B. The simple question was whether the big one earns its size. Before I could answer it, I had to find three mistakes of my own. Once they were fixed, the small model was still ahead.
My yardstick is a private agentic benchmark: each case is a small coding task done in a real sandbox, and it is scored by running whatever the model writes. Nobody grades the prose. The "medium" tier has 40 cases, and from an earlier post I know that running the same model twice can move the score by five cases. So any gap smaller than five counts as nothing.
The first result looked like what I expected: the 27B beat MiniCPM by a wide margin. It also looked like what I expected for the wrong reason.
Mistake one: no model ever got to think
These models can write out their reasoning in a block before answering. A benchmark server lets them do that by default. My engine doesn't: when it builds the prompt, it closes that reasoning block before the model says a word. So in my apps, no model had ever reasoned, and my first benchmark run had measured something users never see.
When I measured both modes, the difference for the small model was huge:
| medium tier, 40 cases, 3 runs each | reasoning | reasoning off (what my apps did) |
|---|---|---|
| Ternary Bonsai 27B | 37 · 39 · 38 | 36 · 35 · 32 |
| MiniCPM 2B | 22 · 25 · 23 | 12 · 11 · 9 |
With reasoning off, MiniCPM nearly always stopped at its second step without touching a single file. With reasoning on, it roughly doubled its score. I also tried giving it four and eight times more room to think, and nothing changed, so space to think was not the bottleneck. Reasoning is now on for that model in the engine.
Mistake two: the right model with the wrong settings
Every model card says how to sample from that model: how much randomness, how many candidate words to keep, and how hard to push back against repeating itself. In production I ran the 27B greedy, always taking the most likely next word, and capped every answer at 512 tokens. That cap is shorter than many of the files the benchmark asks for.
The card asks for something else: a temperature of 0.7, nucleus sampling at 0.8, the top 20 candidates, and a presence penalty of 1.5. With those settings and a longer cap, the 27B went up in the medium tier:
That is a real improvement for the 27B, but it falls inside my own noise threshold. And medium had stopped telling the models apart: Gemma 4 E4B, a fraction of the 27B's size, scored 39 · 38 · 40 with reasoning off, including a perfect 40. At that point the benchmark just has no room left at the top.
Mistake three: the dice were loaded
To carry those card settings into my own engine, I had to read its sampler line by line, and the sampler had been broken all along.
The random-number function that should return a value between 0 and 1 actually returned a value between 0 and 2. Whenever the draw landed above 1, nothing was ever chosen, and the code fell through to the last candidate in the list, which is the least likely one. On top of that, every token got the same seed. The unit test written for the fix confirms it: with the old generator, the least likely token was picked about half the time.
The bug sounds worse than it was. The chat path ran greedy and never rolled the dice, so the only thing using random sampling was an internal voting step. The sampler also applied temperature first, while llama.cpp applies it last. That order changes how many candidates survive the cutoffs. Both are fixed now. The generator is a proper one with a new seed for each token, and there is a test that fails with the old code and passes with the new.
None of the three was in the models. They were in the settings I sent, and in the code that turns those settings into a choice. Whenever a model looked bad, my first question had been "is this model worse?" It should have been "is my app really sending what I think it sends?"
The hard tiers
Medium couldn't separate them any more, so I moved to the hard tiers: 135 cases across six levels, with reasoning off as in my apps, the card settings for each model, and room for long files. As always, I wrote down the decision rule before running anything. I had planned two runs per model, but to save time I cut it to one. Since a single run is noisier, I raised the bar to match: one model has to beat the other by at least 15 cases before I say it is better.
| tier | Gemma 4 E4B | Ternary Bonsai 27B |
|---|---|---|
| high | 5 / 10 | 2 / 10 |
| extreme | 11 / 25 | 13 / 25 |
| insane | 12 / 25 | 11 / 25 |
| varied | 14 / 25 | 11 / 25 |
| brutal | 11 / 25 | 6 / 25 |
| nightmare | 14 / 25 | 14 / 25 |
| total | 67 / 135 | 57 / 135 |
Ten cases is less than fifteen, so by my own rule this is undecided. Still, all of the evidence leans the same way. E4B solved 24 cases that the 27B missed, and the 27B solved 14 that E4B missed. Each level has too few cases to read a trend from it, so I won't. What the table does not show anywhere is the big model pulling ahead on the hard work, which is exactly where I expected it to.
What I changed
Gemma 4 E4B is now the default in every app. The other two are still available, but you have to pick them on purpose. MiniCPM now reasons. The 27B gets its card settings and a longer cap. The sampler is fixed and tested. One caveat on E4B: the benchmark used a standard 8-bit build, while the apps ship the same model in a different runtime format. It is the same model, but not the same file.
The useful lesson isn't about which model wins. I spent days treating a model's score as a property of the model, when part of that score came from my settings and part came from my code. Before comparing two models, I now print what the server actually applied, and I read the sampler. In this case, both checks would have been faster than any benchmark.