Four versions, each better than the last. None had been measured against doing nothing.

An agent's history isn't a list you append to. It's a budget you re-spend on every single turn. Twenty turns into a real session you're holding far more than fits, and something has to go. What goes, and how you choose, is the whole game — and I wrote a post about exactly that, no. 14, with a table of rules and a paragraph explaining each one.

Everything in that post was true. The engine did what I said it did. That turned out not to be the point.

How you get four versions deep into a mistake

The lineage went like this. A first heuristic scorer. Then a better one that beat it. Then one with a reference dimension that beat that. Then one with term weighting that beat that. Four generations, each measured head-to-head against its predecessor, each an honest win.

Look at what's missing from that sentence. Every comparison was internal. The reference point, every time, was the previous thing I had built. It never once occurred to me to ask what a program with no ideas at all would score — because by then I had a scoreboard, and on the scoreboard I was winning.

So I added two lines to the benchmark. One condition that keeps the newest messages until the budget is full and drops everything older. One that scores every line with textbook BM25 against the current question and keeps the best. No cleverness in either.

Which is, word for word, the method I wrote a whole post about in no. 02: take a reference you did not build, run it on the same input, and refuse to trust your version until it matches. I called it oracle-first and treated it as non-negotiable — for a Rust kernel. It never occurred to me that a selector needed one too.

The numbers

The harness generates real agent sessions over real code — every tool result is the genuine output of a real tool on a real project, formatted exactly as the product emits it. The metric is fact recall, and it's objective: no LLM judge. The agent read a definition earlier; later, something asks for it. Did the packed context still contain it? That's the failure that matters — throw something away, then get asked for it.

25 sessions, 174 probes, identical token budget for every condition:

The bottom two rows are the same algorithm. Same formula, same constants, same forty lines of code. The only difference is what it was allowed to score.

That is not the result I set out to write. My first draft said the lesson was rules lose, retrieval wins — and the rules engine really is down there at 10.9 %, barely ahead of throwing everything away. But the browser packer I shipped and documented already had textbook BM25 in it [1]: Robertson's formula, saturation and length normalisation, correctly implemented. It scored 15.1 %.

So the fifty points were never the algorithm. The algorithm was already there. They came from two decisions wrapped around it: the packer trimmed old tool results to a fixed length before scoring them, and it kept or dropped whole messages rather than lines. A tool result is mostly noise with two useful lines in it — score the whole thing and the noise drowns the signal; trim it first and the signal may already be gone.

Having the right algorithm is not the same as putting it somewhere it can work. That's less quotable than "heuristics are bad", and it's the one the numbers actually support.

The rules engine deserves its own sentence, though, because it's the worse story. On a long-term memory benchmark, the pure-heuristic sibling — the one in my command-line agent — scored 2.0 % against 5.8 % for blind truncation. It was actively harmful. Running it made the agent worse than running nothing at all, and it had been in production for months, because "nothing at all" had never been on the scoreboard.

Why the clever version lost

This part is embarrassing in how visible it is. Here's the scoring line, near enough:

score = classifyLine(line)              // hand-written rules → 0.1 … 1.0
      + 0.5 * referenceScore(line)      // → 0 … 0.5
      + 0.4 * queryScore(line)          // → 0 … 0.4  ← the actual question

Read the ranges. The hand-written rules — "if it contains 'error', score 0.95"; "if it starts with $, score 0.6" — could move the score by a full point on their own. What the user is asking about right now entered at four tenths, capped.

The rules outweighed the question. Not by a little; by more than two to one. A line saying error: file not found from thirty turns ago outranked the line holding the answer, forever, because a rule I wrote one afternoon said errors are important. Sometimes they are. The rule can't tell when.

That's what a heuristic is: a guess about relevance, frozen at the moment you typed it, that can never look at the question. Enough of them stacked together stop being a scoring function and start being a personality.

The part I didn't expect

Fixing it meant deleting things, not adding them. Real BM25 with saturation and length normalisation. Scored against the live question, not the opening task. Line granularity instead of whole messages. And no rules at all — not one.

But the piece I keep thinking about is the stopword list. The old engine had one, hand-written, about ninety Spanish words long, to stop "the" and "for" from counting as evidence. I deleted it, and the engine got better.

Because IDF [3] already does that job, and does it properly. Compute term rarity over the conversation you're actually holding, and anything that appears everywhere collapses to near-zero weight by construction. No dictionary. No language detection. Nothing to maintain. A hand-written stoplist only knew Spanish — in an English coding session it filtered nothing — and it aged the moment I stopped editing it. The endogenous version adapts to every conversation on its own, for free, and it works in a language I've never heard of.

That's the shape of the whole lesson, really. Every hand-written list is a small bet that the world will hold still.

The result I did not see coming

Everything so far measures whether the evidence survives packing. It doesn't measure whether the model gets the answer right. So I ran a second, independent benchmark — a public long-term-conversation set, scored with its own published F1 script, so the metric isn't one I invented — with a real model answering 200 questions. Every condition gets the same questions at the same budget: about 8 % of the full conversation.

Read the first row against the last. Giving the model the entire conversation scores 22.56. Giving it a well-chosen 8 % of it scores 28.09.

Retrieval doesn't just survive compression — it beats not compressing at all, by a quarter, on a twelfth of the tokens. I had been treating context management as damage control: how little can I lose. It isn't. The irrelevant 92 % isn't neutral ballast, it's a distraction, and removing it is worth more than everything it contained.

Which reframes the whole exercise. The question was never "how do I fit this in". It's "what should the model be looking at" — and that's a retrieval problem, which is why a retrieval algorithm wins and a pile of rules about what looks important loses.

Why you want both, proven cleanly

Globally, BM25 and embeddings tie — 23.59 against 23.95. Thirty years of word statistics against a modern embedding model, a dead heat. That's worth sitting with on its own.

But a tie can mean two things: they do the same job equally well, or they do different jobs. So I split the questions by whether the question shares vocabulary with the answer:

They swap places. When the question names the thing — an identifier, a path, an error string — lexical wins by eleven points. When it doesn't — "the thing we discussed" — semantic wins by six. The tie was two different tools averaging out. And the fused row takes the better of both columns, which is exactly what a complementary pair should do and exactly why the answer is both, not whichever wins.

How you combine them matters too. Adding weighted scores means calibrating an unbounded BM25 value against a cosine in [-1,1] — a classic source of fragility. Fusing ranks instead of scores [4], ignoring magnitudes entirely, scored higher (28.09 vs 26.36) and deleted the calibration problem. Fewer knobs, better result, again.

Two more lines from that table earn their place. Removing the IDF costs 3.25 F1 — the endogenous term-rarity really is doing work, not decoration. And the heuristic scorer isn't merely useless: switching it on costs 3.02 F1 versus leaving its weight at zero. It was subtracting the whole time.

The finding that actually matters

Everything above is at one budget. Real sessions aren't one budget — they're a long context under varying pressure. So I swept it, from a budget sixteen times smaller than the conversation down to one that nearly fits:

Look at the middle column. 13.2 %, 13.2 %, 13.2 %, 13.2 %.

Give the shipped engine ten times more room and it recovers exactly nothing extra. It isn't merely worse — it's deaf to the budget. And the reason is a single design decision I'd have defended as sensible: it trimmed old tool results to a fixed length, and it kept or dropped whole messages. Both of those throw information away before anything is scored. You cannot recover what you discarded before you measured whether it mattered. More budget can't help, because by the time there's room, the thing is already gone.

That's a nastier class of bug than being slow or being wrong. It's a system that looks like it's working — it packs, it fits, it never errors — and is structurally incapable of getting better.

The constants were the same disease, one floor up

Having deleted all the content heuristics, I still had a fistful of magic numbers: keep 55 % of the budget for recent turns, penalise redundancy at 0.5, consider 600 candidates. Hand-picked, exactly like the rules had been.

The sweep caught one of them red-handed. I keep a mild preference for recent lines. Under heavy pressure it's worth +11 points. At three-to-one it costs 8. Same constant, opposite sign — because they're different problems: when almost nothing is relevant, newest is the only bet left; when there's room, biasing toward new just evicts relevant old material, and keeping contiguous stretches beats scattering.

My first fix was to fade the weight with pressure. It did nothing — measured, 0.048 and 0.15 make identical decisions, and only zero changes anything. As an additive term, recency is effectively binary: any positive value lets a new irrelevant line outrank an old relevant one. The knob I was tuning had two settings and I'd been turning it like a dial.

So it stopped being a weight. Relevant lines now occupy a strictly higher band than irrelevant ones — no amount of recency can cross — and recency only orders the irrelevant ones, and only when pressure is high enough that ordering them is all that's left. One rule, derived from the sweep, no dial. The result matches or beats the best hand-tuned variant at every single point of the curve, with one configuration.

Which is the same lesson as the stopword list, and the same lesson as the rules: every constant is a bet that conditions won't change. Measure the condition and the constant disappears.

The fix that made it affordable, and where it came from

A hybrid that scores twice on every turn sounds expensive, and the naive version is: my old engine recomputed term statistics over the entire history on every single message, so cost grew with the square of the conversation. At eighty turns that was already the slowest thing in the loop.

The fix is to move the thinking to write time instead of read time. Each tool result gets indexed exactly once, as it lands — terms counted, block encoded, cached by content. Every later turn only scores against what's already there. A line seen at turn three is never re-encoded at turn eighty. That single change is what turns "run two retrievers per turn" from a nice idea into something that fits in an agent loop, and it's the same principle I'd already leaned on elsewhere in this notebook: in no. 12 the four gigabytes of model weights are paid once and cached forever, because the alternative — paying on every use — is simply not a product.

The other end of the same problem

Jesús Cardeñosa spent years on UNL, an interlingua: encode a document once into a formal, language-independent semantic representation, then retrieve over that instead of over the words. It's the opposite pole from BM25 — explicit meaning versus pure surface statistics — and the same problem attacked from the other end, two decades before anyone was compressing an agent's context.

Which makes my stopword result funnier than it deserves to be. Deleting that hand-written list gave me language independence by accident, through the cheapest mechanism available, landing on a design goal that line of work pursued deliberately and properly. And the tie in the table above — lexical and semantic, dead level, each covering the other's blind spot — is a real data point for that old argument: after all this time, neither pole wins alone.

I'd add, quietly, that he pointed me at term weighting for exactly this kind of problem a long time ago, and I filed it under classical information retrieval and moved on. Everything above is that idea, arriving late and with a benchmark attached. Which is really the same lesson as the rest of this post, in a softer form: a good reference point is often already available — in the literature, or in a conversation — and the work is noticing that you need one.

The bug retrieval cannot fix

Everything above is about finding the right lines. Here is a failure where finding them perfectly is still wrong.

An agent reads builder.js at turn 3. At turn 20 it edits the file and reads it back. Both results are now in the history, and they differ in one number. Ask what that number is, and BM25 does its job flawlessly: it retrieves the lines that match the question. Both of them. The old value and the new one, side by side, with nothing to say which is which.

That isn't inefficiency. No amount of relevance fixes a fact being false. I measured it with a probe that reads, edits, and re-reads: the stale value survived 8 times out of 8, at every budget.

The fix is borrowed from somewhere unglamorous — temporal databases [11], where a superseded row isn't deleted, it's given a validity window and demoted. Here the timestamp is the turn number and the identity is the tool call's target: when a later call touches the same file, the earlier result stops being current. It gets pushed below everything relevant, so it only appears if nothing better exists. Stale: 8/8 → 0/8. Current: 8/8. Fact recall unchanged.

And here I caught myself doing it again. I wrote that demoting rather than deleting means "if you ask what the file used to say, it's still there" — because that's what the data structure does, and it sounded right. Then I measured it. Asked explicitly for the previous value, the old version comes back 0 times out of 8. It is in the pool and it is never retrieved: once demoted it sits among hundreds of zero-score lines, and which of those come back is decided by a diversity pass, not by the question. Reachable is not retrievable. The distinction between "no longer true" and "never existed" is real in the data structure and not yet real in the behaviour, and I'd have shipped the slogan — in this post, about exactly this failure — if I hadn't checked.

And underneath it was a second bug, which is the one I'd have never found by reading the code. Demoting the stale lines did nothing at all — because a separate mechanism pins any line carrying a term from the current question, so it can never be evicted. The old version carries the same identifier as the new one. It pinned itself. A safety net designed to protect the answer was holding the wrong one in place.

This one is embarrassing in a specific way: no. 15 describes a folder watcher that tracks files "by name and mtime — so re-saving a file reprocesses it". Identity plus timestamp, so that a newer write invalidates the older read. I had solved this exact problem one floor down, for files on disk, and never thought to apply it to the conversation.

What these numbers don't say

Two benchmarks, two different questions. One asks whether the evidence survived the packing; the other whether a real model, handed that context, then answers correctly. They agree — which is the only reason I trust either of them. But both lean heavily on English, and the second is licensed for research. It can tell me the engine works. It can't ship with it.

The ablations were their own kind of humbling. Redundancy control is worth about two points on agent sessions and nothing at all on conversation — dialogue has no near-duplicate lines to collapse, and I had assumed it would help everywhere. Two other mechanisms I was fond of turned out to add nothing measurable. Term rarity, measured against the live question, is doing nearly all of the work. The rest is decoration, and I'd have kept every piece of it forever if I hadn't switched them off one at a time.

Two are still open. The cheap one first: dates. "Yesterday", written at turn 3, is a lie by turn 40 — and no retriever can repair that, because it isn't a retrieval problem. It has to be resolved to a calendar date when it's written or not at all. The harder one is aggregation: "how many files did you touch?" has no answer in any single line, only spread across forty of them, so top-k cannot reach it by construction. Both need work done at write time, not read time.

And one trade-off I'd most like settled. The packer now stops when it runs out of relevant material rather than when it runs out of tokens — which at a large budget saves 60 % of the tokens and costs 9 points of recall. Whether that's a good trade depends on something neither benchmark measures: does a shorter, cleaner context produce a better answer? Everything above suggests it does. Nothing above proves it.

And no. 14 already contained the sentence "BM25 doesn't understand that 'the thing we discussed' refers to the vault". I had identified the limitation correctly, published it, and then left it unmeasured for months. Knowing where your thing is weak is not the same as knowing how weak it is.

As I already pointed out here — three times

The uncomfortable part is that this notebook already contains the answer, written by me, more than once:

• no. 02 — oracle-first. Never trust your implementation until it matches a reference you didn't write. I applied it to the seventh decimal on a Rust kernel, and not at all to the thing choosing what the model gets to see.
• no. 09 — I trained on the benchmark and it got worse. Optimising against a target that isn't the real one makes things worse while every number you watch goes up. That is exactly this post, with the benchmark swapped for my own previous version.
• no. 06 — I optimised the GPU and it came out 20× slower. The obvious improvement, measured, was a catastrophe. Same shape: obvious is a hypothesis, not a result.

Three posts, one lesson, and I still walked into it — because each time the failure wears a different costume. In no. 06 it was a kernel, in no. 09 a fine-tune, here a scoring function. What they share isn't the domain: it's that the reference point was chosen by the same person who wrote the thing being measured. There's a version of no. 11 in here too — a system producing fluent, plausible output while ignoring the input it was supposed to be using.

Writing a lesson down is not the same as having learned it. The only thing that actually catches this is a number that comes from outside.

References

1. Robertson, S.E., Walker, S., Jones, S., Hancock-Beaulieu, M.M. & Gatford, M. (1994). Okapi at TREC-3. Proceedings of the Third Text REtrieval Conference (TREC-3), 109–126. — the BM25 ranking function used throughout this post.
2. Robertson, S.E. & Zaragoza, H. (2009). The Probabilistic Relevance Framework: BM25 and Beyond. Foundations and Trends in Information Retrieval, 3(4), 333–389. doi:10.1561/1500000019 — modern formulation, and the justification for the k1 (saturation) and b (length normalisation) terms whose absence broke the first engine.
3. Spärck Jones, K. (1972). A statistical interpretation of term specificity and its application in retrieval. Journal of Documentation, 28(1), 11–21. doi:10.1108/eb026526 — the origin of IDF: rarity is informativeness. Deleting a hand-written stopword list works because this does the job properly.
4. Cormack, G.V., Clarke, C.L.A. & Buettcher, S. (2009). Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods. SIGIR '09, 758–759. doi:10.1145/1571941.1572114 — why fusing ranks beats fusing scores, and why no calibration is needed.
5. Carbonell, J. & Goldstein, J. (1998). The use of MMR, diversity-based reranking for reordering documents and producing summaries. SIGIR '98, 335–336. doi:10.1145/290941.291025 — the redundancy penalty applied at selection time.
6. Xiao, G., Tian, Y., Chen, B., Han, S. & Lewis, M. (2024). Efficient Streaming Language Models with Attention Sinks. ICLR 2024. arXiv:2309.17453 — the first tokens absorb a large share of attention mass; the empirical basis for reserving the head unconditionally rather than scoring it.
7. Maharana, A., Lee, D.-H., Tulyakov, S., Bansal, M., Barbieri, F. & Fang, Y. (2024). Evaluating Very Long-Term Conversational Memory of LLM Agents. ACL 2024. arXiv:2402.17753 — the long-conversation benchmark and its own F1 script, used unmodified. Licensed CC BY-NC: used to measure, not shipped.
8. Khattab, O. & Zaharia, M. (2020). ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT. SIGIR '20. arXiv:2004.12832 — late interaction, the stronger retrieval family this engine does not use, and the honest ceiling on what is reported here.
9. Wang, L., Yang, N., Huang, X., Yang, L., Majumder, R. & Wei, F. (2024). Multilingual E5 Text Embeddings: A Technical Report. arXiv:2402.05672 — the embedding model behind the optional semantic side (MIT-licensed ONNX conversion, run in-browser).
10. Snodgrass, R.T. & Ahn, I. (1985). A taxonomy of time in databases. SIGMOD '85, 236–246. doi:10.1145/318898.318921 — valid time versus transaction time. The reason a superseded tool result is demoted with a validity window rather than deleted.
11. CodeComp: Structural KV Cache Compression for Agentic Coding (2026). arXiv:2604.10235, §4.4 — the closest published work: it uses query–symbol overlap, but as a binary protection flag over a ranking that is still attention-based. Measured here against a graded formulation: F1 8.03 vs 23.59.

All measurements in this post are reproducible from the harness in the linked repositories. Where a number could not be verified on this machine, the post says so.

The engine is Apache-2.0 and lives in elffuss core, shared by elffuss-code and elffuss-claw. Previous: your folder never leaves · back to the index.