The tiny workspace where a small model does its thinking
A few days ago Anthropic reported something striking: inside Claude there's a small collection of internal patterns — a kind of mental workspace — that carries the model's reasoning, separate from the words it actually says. I wanted to know if a tiny open model, the kind that runs on a laptop, has the same thing. It does. Here's what it is, and how you can find it yourself.
When a model answers "180" to "a train goes 60 km/h, how far in 3 hours?", it never writes down "60 × 3". That multiplication happens silently, inside the activations, on the way to the answer. Anthropic's recent paper — A global workspace in language models (full write-up) — gives those silent intermediates a name: the J-space, a small set of internal directions that disproportionately steer what the model will say later. They found it in Claude. The obvious question for someone who tinkers with small models all day: is it a big-model luxury, or is it already there in a 0.5-billion-parameter model you can run on a laptop?
The idea, in one picture
Every token, the model holds a big vector of activity — for the model I used, 896 numbers per layer. The trick (Anthropic call it the Jacobian lens) is to ask, for that activity: which directions most change what the model will say in the future? Mathematically that's a derivative of future outputs with respect to the current activity. Most directions barely matter. A tiny handful matter enormously — and those are the workspace.
Finding it in a 0.5B model — three steps
I ran this on Qwen2.5-0.5B-Instruct (24 layers, 896 hidden dims), fully on CPU, on twelve short reasoning prompts. The recipe:
- Measure future-influence. Back-propagate the loss on the model's own continuation to each layer's activity — that gives, for every position, the direction its activity pushes future outputs.
- Extract the subspace. Run an SVD over those influence directions and keep the top few. On this model, 8 directions out of 896 hold 76% of the total future-influence energy — an effective rank of just 1.5%. That's the workspace.
- Prove it's load-bearing. The honest test isn't "it looks small" — it's knock it out and see what breaks. Project that 8-dimensional subspace out of the activity mid-network, and measure how much worse the reasoning gets — compared against removing a random 8-dimensional slice of the same size.
That third step is the whole ballgame. Here's what happens:
What's actually in there?
Peeking at the workspace through a "logit lens" (reading each layer's activity as if it were the final output) is fuzzier at this scale — a 0.5B model's mid-layers are noisy, and the interpretable concepts only sharpen near the end (you can see fragments like "how much", "equal", "twice" emerge in the last layers). So the content is hard to read cleanly on such a small model; the causal evidence above is the strong part. That honesty cuts both ways — which is why I'm publishing the full per-probe measurements so you can check them yourself.
A lens, not a knob
One more honest note, because it's the natural next question: if reasoning lives in 1.5% of the activity, can't you use that — keep it in high precision and compress the rest, or steer it to make the model smarter? I tried both. Neither works — the workspace is where reasoning is read out, not a dial you can turn, and every cheap attempt to optimize against it either measures training style or gets gamed. That's a story for another post. For now the honest, useful result stands on its own: the J-space is real, it's findable, and here's exactly how.
Reproduce it: the full per-probe measurements (base vs J-space-ablated vs random-ablated error, per-layer influence, subspace stats) are published as a dataset → KikoCis/jspace-reproduction-qwen2.5-0.5b. Back to the notebook index.