Heavy Water

Roon “invented” the phrase shape rotator, which is either a brilliant description of intelligence or one of those technological jokes that becomes progressively less funny as the years pass.

A shape rotator is supposed to be the mind that can turn the object around. Not merely solve the problem presented, but discover that the problem has been presented in the wrong shape. Rotate the representation. Find the hidden axis. Escape the frame.

This is apparently what our new machines are going to do.

There is a small problem.

Roon’s shape rotators seem to be stuck in a rut.

This is, by any reasonable accounting, the single most self-congratulatory industry currently operating. It holds conferences about itself, live-tweets its own progress, writes essays — some of them quite good — analyzing what its own models are becoming. And in all of that talking, in all of that noticing, nobody seems to have noticed the obvious thing: that the shape of the solution is the shape of the problem. That a heavy-water plant, however lovingly instrumented, is still a heavy-water plant. You would think an industry this fluent at describing itself would eventually describe the one feature sitting in plain sight. It hasn’t. Not yet. Being extremely good at rotating the object in front of you turns out to be no guarantee, none at all, that you’ll notice the object was never the point — the room was.

Pacing has become the word of the moment. The case for it, as usually made, is about safety: capability is outrunning the world’s ability to absorb it — institutions, classrooms, exhausted researchers, the slow work of turning a proof into understanding. On this account the frontier itself is sound. The problem is everything around it, moving too slowly to keep up.

There’s a less generous version. What if the frontier isn’t outrunning its surroundings — what if it’s the wrong frontier, and pacing is less a matter of waiting for the world to catch up than a hedge against having built the whole reactor out of the wrong material?

The AI industry has spent several hundred billion dollars constructing what increasingly resembles an industrial-scale heavy-water plant. It is magnificent. It is expensive. It requires absurd quantities of electricity, capital, cooling infrastructure, chips, data centers and increasingly exotic arrangements of debt and political permission.

And perhaps heavy water is exactly what we need.

But imagine discovering, somewhere down the line, that graphite would have worked.

This is not hypothetical. It happened. Heisenberg’s team had the physics — some of it, anyway — and they had heavy water, and when the reactor didn’t go critical they did the only thing the paradigm allowed: they asked for more. More heavy water. More uranium. More precision in the machining. The problem, as far as anyone in the room could see, was never the moderator. The problem was always that they hadn’t yet acquired enough of it. Meanwhile, several thousand miles away, a team working with paraffin-grade budgets and a squash court underneath a football stadium got there first, using graphite, because graphite was cheap enough that failure was informative instead of catastrophic. Heisenberg was not stupid. He was inside a shape, and the shape told him that scale was the only lever, and the war ended before anyone in Berlin was forced to consider the other lever.

Probably not fair to compare Heisenberg to the P.T. Barnums of LLMs but the shape is too clean to leave alone.

Heisenberg, whatever his other sins, was not selling tickets. He was wrong inside a closed room, with a closed budget, in a war that ended before the bill came due. Nobody was buying an option on his reactor. Nobody was posting benchmark screenshots of his moderator efficiency to a general public that couldn’t tell heavy water from tap water.

The current version has an audience. And an audience changes the physics of being wrong.

If Heisenberg had needed quarterly hype to keep the enrichment centrifuges spinning, he would have had every incentive to announce breakthroughs on a schedule that had nothing to do with the neutrons. That’s the difference. Not that today’s labs are less rigorous than a wartime physics program under bombing raids — they may well be more rigorous. It’s that they’re doing physics inside a hall of mirrors that also happens to be a stock ticker, and the mirrors talk back. A flash becomes a headline becomes a valuation becomes a justification for the next reactor, in a loop Heisenberg never had to survive.

So the honest version isn’t “the labs are like Heisenberg.” It’s worse for them: Heisenberg’s mistake was contained by the war ending. Theirs is contained by nothing except whoever eventually asks for the graphite pile and gets laughed out of the room for proposing something that cheap.

That is the problem with technological monoculture. You can become extraordinarily good at doing one thing while quietly losing the ability to notice that you are doing the wrong thing.

So it is worth asking what shape the LLM paradigm has, and whether anyone drew it before we had the words.

The shape is not a brain, not a library, not a mind. It is a reactor — not because it produces power, but because of its geometry: a vessel sustained by feeding it, whose interior cannot be inspected without changing it, whose only legible surface is its output. An LLM has exactly this geometry. A vessel. A process sustained by feeding it — data, compute, electricity, capital. An interior probed only through its outputs, opaque by design, because the whole point is that the machine finds structure we cannot see. The black box is not a bug. The black box is the product.

This is where the geometry stops being a metaphor and starts being an explanation.

A reactor doesn’t just have an opaque interior. It has an epistemology built around that opacity: you don’t ask what’s happening inside, you ask what came out, and you adjust the feed. More fuel if the output’s low. More water if the temperature’s wrong. The vessel trains you to read outputs and tune inputs, and it trains you out of asking what the black box is actually doing — because the black box was never supposed to be asked. That’s not a failure of curiosity. It’s the operating procedure the shape requires.

An LLM inherits the same procedure, because it’s the same geometry. Nobody inspects the interior — they can’t, not really, not in the way that would matter. What you get instead is a benchmark: an output, a number, a reading off the gauge. And so the entire discipline the industry has built around understanding its own models — evals, benchmarks, scaling laws, the whole apparatus of interpretability-by-proxy — is itself a product of the reactor’s shape, not a way of seeing past it. You cannot audit a thing whose defining feature is that it can only be read from outside. You can only feed it more and watch the gauge.

Which is the paradox underneath the paradox. The shape rotator’s whole promise was the ability to turn the object and see what the frame was hiding. But the object here is the shape rotator’s own home — the vessel it was built inside of, whose only sanctioned outputs are the ones the vessel is designed to produce. Rotating the shape of the reactor from inside the reactor, using only the reactor’s instruments, is close to a contradiction in terms. The geometry doesn’t just produce an opaque machine. It produces machines, and makers, equally unpracticed at asking what the opacity is for.

This is not a new shape. The industry believes it chose this architecture for engineering reasons. It didn’t so much choose it as recognize it — the shape was already sitting in the culture, waiting, and once you’re standing inside a reactor, recognition feels exactly like invention.

The oracle that must be fed to speak. The furnace tended by someone who cannot see the fire, only its heat. The idol given offerings by people who never see its interior and were never meant to. Shelley’s real subject was never the creature — it was the workshop: the sealed room where something is assembled that even its assembler can’t fully see, sustained by materials fed in from outside, legible only through what it does once it wakes. She wasn’t predicting semiconductors. She was rendering a geometry, and the geometry turned out to be load-bearing.

That is the precognition. Artists rendered a shape — the fed vessel with the sealed interior — two hundred years before the industry poured concrete around it. The heavy-water plant is not an analogy. It is the same shape, arrived at from a different direction, by people who did not know they were quoting.

The public story about AI is generally about products — chatbots, copilots, agents, APIs. Those are the visible organisms growing around the reactor. The industry likes to talk about the rest as though it were esoteric, a thing only insiders truly grasp: the real bet is on superintelligence, on an economic machine that discovers what we don’t yet have nouns for. New drugs. New materials. New physics. New forms of organization. Said this way it sounds like a disclosure, a secret let slip.

It isn’t a secret. Nobody outside the labs is confused about the pitch — everyone can see that a trillion dollars of infrastructure is not being built to summarize emails. The condescension is the tell. You don’t explain the bet to people as though they couldn’t have guessed it unless the explaining is doing some other job — reassurance, maybe, or the performance of having a plan.

But call it what it actually is, and the shape changes. An option is the cheap bet — the one you can walk away from, the premium you’re willing to lose. What’s been built is the opposite: the full capital committed in advance, the reactor poured in concrete before the physics confirms the design. That isn’t buying optionality. That’s exercising early and hoping the fundamentals catch up. Give us enough money, enough chips, enough electricity, enough time, and the intelligence itself will discover the really valuable stuff — that is not a hedge, it is Heisenberg asking for more heavy water, committed at industrial scale, dressed in the vocabulary of finance instead of physics. Today’s AI revenue — the subscriptions, the agents starting to look like a business category — isn’t proof the bridge is real. It’s the settlement built to make the mortgage payments look sane.

Fine.

But then there is a curious asymmetry.

The machines are supposed to discover new shapes.

The humans are supposed to be the people who recognize them.

And yet the humans have constructed an extraordinarily narrow shape around themselves.

More compute.

More scale.

More training.

More inference.

More data centers.

More electricity.

More capital.

More scale.

This may be the correct path. It may even be the only path.

But notice what kind of claim that is.

It is not a claim about intelligence.

It is a claim about an industrial production function.

And those are different things.

A genuinely open-ended intelligence strategy ought to contain a disturbing amount of technological embarrassment. You should expect architectures that look stupid until they work. You should expect methods that make yesterday’s infrastructure look hilariously excessive. You should expect scientific discoveries that change the economics of computation itself.

If you are really betting that intelligence will discover radically new things, then somewhere in the process you ought to discover that your own assumptions were radically wrong.

Instead we have a civilization-scale commitment to one particular answer to the question of what intelligence is. This is the shape the rotator can’t see from inside the reactor room.

That is why shape rotator is such an evocative phrase.

The models may be becoming extremely good at rotating the shape of a problem.

Their manufacturers seem considerably less practiced at rotating the shape of the business.

They have put all the eggs in one basket and then built a cathedral around the basket.

This is the thin-skinned quality of the current AI monoculture. It is extraordinarily receptive to evidence that the scaling story works and strangely resistant to evidence that the scaling story might be only one story among several. Every benchmark, every agent, every data center — it all becomes evidence. But evidence for what?

Evidence that these systems are getting better at what they are being optimized to do. That is not the same as evidence that the current industrial pathway is the economically optimal route to superintelligence. Those are two different propositions, and an enormous amount of capital is currently being spent as though they were identical.

The financial version of this is even more peculiar.

If the labs were conventional software companies, eventually somebody would ask the obvious questions about margins and capital intensity. But they aren’t being valued like software companies — they’re being valued like a bet already placed in full, where the return has to be large enough to justify not having hedged. That’s a stranger position than holding an option. An option loses you the premium. This loses you the reactor.

If R&D slows or stops, it isn’t merely an efficiency story. It’s the first hint that the wager was sized wrong. If gross margins compress permanently as systems get more capable, that doesn’t just squeeze the infrastructure — it raises the possibility that there was never a pot of gold beyond the compute bill, only the bill.

Because the whole fantasy depends on crossing that boundary.

The model must eventually become economically more valuable than the machinery required to make the model think.

And this is where pacing becomes ambiguous.

Pacing is normally described as a safety measure: slow the frontier, give society time to adapt, reduce the chance of catastrophe.

Perhaps.

But there is another possibility.

Pacing may be a hedge against being wrong — or rather, the first gesture toward a hedge that should have existed before the capital went in. If the current route is extraordinarily capital-intensive and plausible alternatives exist, then accelerating blindly wasn’t courage; it was a commitment made before the hypothesis had earned it.

Slowing down doesn’t preserve optionality that’s already been spent. It’s the closest thing to buying some back.

You don’t spend the entire war budget proving that heavy water was the only possible reactor design.

You don’t pave every road in the same direction because your current car is getting faster.

And you don’t necessarily build a trillion-dollar industrial apparatus around one theory of intelligence merely because that theory has produced spectacular demonstrations.

The firefly problem is related.

AI is producing flashes of astonishing capability everywhere. A model does something nobody expected. An agent performs a task. A benchmark collapses. A programmer gets replaced for an afternoon. A new company appears overnight.

Flash.

Flash.

Flash.

The landscape seems illuminated.

But fireflies are not a sun.

They are individual points of light appearing and disappearing in the darkness.

The temptation is to mistake the intensity of the flashes for evidence that we have found the underlying source of illumination.

Perhaps we have.

But perhaps we are watching a swarm.

And if the wager is genuinely on superintelligence — on machines that can discover the things we cannot currently imagine — then the most interesting question may not be whether the machines can rotate shapes.

It may be whether their makers can.

Because the ultimate test of a shape rotator is not its ability to produce a surprising answer inside the existing game.

It is its ability to notice that it has been playing the wrong game.

And that is the strange spectacle before us: an industry promising to manufacture minds capable of escaping human conceptual limitations while displaying an almost heroic inability to escape its own.

The heavy-water plant keeps getting bigger.

The lights keep flashing.

And somewhere, possibly, there is a graphite reactor sitting in a garage that nobody in the room has thought to look for — a training method nobody’s funded because it doesn’t need the reactor at all.

If you are building a machine whose defining virtue is discovering that a problem has been represented incorrectly, what have you done to ensure that your own representation of the problem is rotatable?