spatial · 3D understanding no. 11

My AI wrote a flawless floor plan of a building it wasn't looking at.

I'm building a system that takes a raw 3D scan of a real building and writes out its structure — where the walls, columns, doors and windows are. The part that writes the plan is a small language model, and it produced beautiful, syntactically perfect plans. There was one problem: they described a building it wasn't looking at. Here's how I proved the model was ignoring the scan, why it happened, and the simpler thing that fixed it — with the numbers on real construction sites.

1 Aug 2026·~8 min read 3Dpoint cloudshonest

The task, in plain words: point a scanner around a building, get millions of 3D points back, and turn that cloud of points into something an architect can use — a structured plan that says "there's a wall here, a column there, a door in that wall." Two moving parts do the job. An encoder looks at the 3D points and turns each one into a rich numeric fingerprint. Then a small language model reads those fingerprints and writes the plan, one token at a time, in a tiny formal language: wall_0=Wall(…), column_1=Column(…), and so on.

It's a lovely design — the same "predict the next token" machinery that powers chatbots, pointed at geometry. And on paper it worked: the model emitted clean, well-formed plans every time. Then I looked at what it was emitting.

The plans were perfect. They were also always the same.

Scene after scene, different buildings, the model wrote almost the same plan — the same few walls at nearly the same coordinates, whatever it was shown. It had learned the shape of a correct answer and was reciting it. Watch it happen, and watch the telemetry on the right: the gauge for "is it actually reading the 3D scan?" barely moves.

That last line in the panel is the test that settled it. I ran the model normally, then ran it again with the 3D fingerprints zeroed out — the geometry deleted, nothing but blanks where the scan should be. The output barely changed. A system that produces the same answer with and without its input isn't using its input. It had memorized the template and was narrating it with total confidence.

This is the failure mode that makes fluent models dangerous as measurement tools: fluency is free. A next-token model will happily produce a perfectly-formatted, grammatical, plausible answer that owes nothing to the evidence in front of it. The polish is not proof. If the output looks flawless, check whether deleting the input changes it.

So the encoder must be blind, right? Wrong.

My first instinct was that the encoder — the part that looks at the points — was the weak link, and I'd need to rebuild it from scratch (expensive, slow). Before doing that, I ran the cheapest possible check: are the encoder's fingerprints any good on their own?

They're excellent. Take the fingerprints for a room and ask a trivial classifier — nothing fancy — "is this point part of a wall, or part of the floor/ceiling?" It gets it right 99.8% of the time. Ask it to spot column points, and its raw signal is 8× stronger than chance. The information needed to locate the structure was right there in the fingerprints the whole time.

Same encoder, same fingerprints. The language model on top writes a fluent plan that owes nothing to the scan. The per-point path on the bottom asks every point what it belongs to, then groups the answers into shapes — and recovers the real structure. The bottleneck was never the eyes; it was the mouth.

The fix: stop asking the sentence, ask each point

If a next-token model reciting a template can't be trusted to use the fingerprints, then don't route the answer through a sentence at all. Instead, attach a tiny classifier that labels every single point — wall, column, door, window, or background — directly from its fingerprint. Millions of dense, cheap yes/no questions instead of one fluent paragraph. Then group the labeled points into instances and fit the geometry (a wall is a line of wall-points; a column is a compact cluster).

It's a humbler design, and it doesn't hallucinate, because there's no template to fall back on — each point can only answer for the geometry under it. Here's the honest head-to-head on held-out construction sites the system never saw during training:

the result On real construction scans it had never seen, the fluent language model localizes essentially nothing (~0 on every element). The per-point classifier on the same fingerprints recovers the structure — 0.46 macro-F1 across walls, columns, doors and windows — decoder-free. The signal was always in the features; the generator was throwing it away.

The second trap: I was measuring walls with the wrong ruler

Here's a confession, because this notebook only earns trust if I show the mistakes too. For a while, my own numbers told me the per-point path had failed on walls — it scored below the useless language model. I almost abandoned the whole approach on the strength of that.

The ruler was the problem, not the walls. The metric I'd inherited scores a predicted wall by how much its thin rectangle overlaps the true one. But walls are thin — a few centimetres across — so a prediction that's perfectly placed but a hair off sideways gets almost no overlap credit. Under that ruler even a perfect answer caps out low, and the difference between "found the wall" and "missed entirely" vanishes into the noise.

The identical prediction. On the left, the overlap ruler punishes a sideways hair's-breadth as if the wall were missed. On the right, the centerline ruler asks the question that actually matters for a wall — "is it the same line, running the same way?" — and sees a near-perfect hit. The columns dodged this trap because they were scored by matching centres, not areas. Walls needed the same courtesy.

Switch to a ruler that fits the shape — for a wall, does the predicted centerline land on the true one — and the picture flips completely:

Same predictions, both times. The strict-overlap ruler said the per-point walls were worthless (0.006) and the language model marginally "better" (0.075) — a comparison of two kinds of noise. The shape-appropriate ruler reveals the truth: the per-point path lands walls at 0.43 with no ground-truth help, against the language model's 0.04, and an 0.90 ceiling if the grouping were perfect. The lesson from post no. 09 came back around: if a result surprises you, suspect the ruler before the world.

The honest edges

This is a working notebook, so here's what doesn't shine yet:

Doors and windows are the weakest. They're openings — absences of wall — so there's little geometry to grab onto, and they land around 0.39–0.44. That's genuinely useful and it's infinitely better than the language model's zero, but it's the soft spot, and a dedicated "hole in the wall" detector is the obvious next move.
Walls are limited by grouping, not by seeing. The gap between the blind 0.43 and the 0.90 ceiling is entirely in how points get grouped into separate walls (corners fuse two walls into one). I tried a fancier learned grouping head and — honestly — it did not beat simple line-fitting. The lever here is better geometry, not a bigger model.
These are localization metrics. "Centre-match" and "centerline-match" reward putting the element in the right place, not reproducing its exact extent to the centimetre. I'm measuring "did you find it and place it," which is the honest claim to make right now.
It's a small classifier on frozen features — no giant training run behind the win. The encoder that produces the fingerprints was pretrained separately; the point of this whole exercise was that it was already good enough, and the generator was the thing wasting it.

The takeaway

Two lessons, and they're the kind you only learn by getting fooled first. One: a fluent generator will narrate a memorized template with total confidence — so before you trust its output, delete its input and see if the output flinches. Two: supervise where the signal is dense. The fingerprints knew what every point was; asking each point directly recovered the structure that routing everything through one eloquent sentence had thrown away. And a coda from an old friend of this notebook: when the numbers say your good idea failed, check that you're measuring it with a ruler that fits its shape.

Testbed: Rohbau3D, a public CC-BY dataset of real construction-site scans (columns, walls and all). All figures above are on sites held out of training. Back to the notebook.