latent reasoning

Remind Me Again Why Large Language Models Won’t Kill Us All

September 2026.

In April 2023, four months after ChatGPT arrived, I published an essay called Remind Me Again Why Large Language Models Can’t Think. It took the standard objections of the moment — no sensory grounding, pattern-matching is not understanding, these networks cannot represent relations precisely, hallucination betrays the absence of a world model — and argued that none of them closed the case. That was a minority position by a wide margin: Gary Lupyan had polled 170 researchers and 80% were strongly of the view that language models definitely could not think. I had been in that majority myself until shortly before writing. Nobody at a frontier lab argues that now. They argue about whether anyone can watch these things reason. That is a long way for a pendulum to travel in three years, and it went past the middle without stopping.

What I got wrong was the route. I expected progress to come from better internal representation, and what happened instead was three years of making models write vastly more down, out loud, in English, because narrating the work turned out to be the cheapest way to improve it. That accident is what AI safety now rests on, and it is being undone deliberately. The reassurances offered against worrying about it are, almost without exception, claims about what has not happened yet.

The argument runs in nine steps.

Safety is premised on monitoring words

A 2025 position paper with more than forty authors from OpenAI, Anthropic, Google DeepMind, the UK AI Safety Institute, Redwood Research and Apollo rests on a single observation: models are forced to externalise their reasoning, so we get to read it. Nobody designed that. It is a side effect of arithmetic, and it is the reason anyone can claim to be watching these systems at all.

The arithmetic is this. A language model picks each word from a vocabulary of roughly 130,000. One choice out of 130,000 is about seventeen bits, in the way one choice out of two is one bit. Between finishing one word and starting the next, seventeen bits is everything that crosses.

The real figure is smaller still, because English is enormously redundant. Shannon's guessing experiments put ordinary literary English near one bit per letter, about 75% redundant, and Cover and King's gambling estimate converged on 1.3 bits per symbol. Measured against real corpora, a good model compresses text to well under a bit per byte. The pipe is narrower than the nominal number suggests, and language works anyway.

Inside, the model holds several thousand numbers describing where it has got to. Picking a word throws almost all of them away and hands the next step a single dictionary entry. So anything needing a long chain of dependent steps has a storage problem: what was worked out at step three cannot be carried in mind to step nine, because the gap between steps is too narrow.

It has to be written down to survive the trip.

Reading it is not the same as reading the reasoning, even now. Plant a hint that a model demonstrably uses, and it mentions that hint well under a fifth of the time. The transcript is what the safety case has, not what the model did.

How much a person can transmit

The same constraint shaped human language, in a far more severe form.

Coupé and colleagues measured seventeen languages and found human speech centred on 39.15 bits per second with a standard deviation of 5.10, with syllable rate and information density trading off against each other so that wildly different languages land in the same narrow band. Whatever we are doing, we all do it at roughly the same speed.

Meanwhile Zheng and Meister put the whole of human behavioural throughput at about 10 bits per second, against sensory systems gathering data at around a billion bits per second, a ratio of a hundred million to one. Everything you have ever known has left your head through an aperture that narrow, and every idea you have received arrived through somebody else's.

We compress because we have no choice. There is one channel out of a human head and it is astonishingly slow. That is why language evolved to be so good at compression, and why we are impressed by people who are good at it.

A machine has the same shape and none of the constraint. Its internal state is thousands of numbers per step. Anthropic trained a dictionary of 33,554,432 features on a single middle layer of one model and found that fewer than 300 were active on any given token: tens of millions of available directions, a few hundred lit at once. Nothing physical requires any of that to be rounded down to one word from a dictionary before being passed to the next step, or to another machine. We compress because we have to. Machines do it because we built them to copy us.

Removing the rounding step

Picking a word is a rounding step. Latent reasoning removes it: instead of rounding those few thousand numbers to the nearest dictionary entry, the model passes them forward as they are, with the same layers, attention and cache. The only change is that thousands of numbers cross between steps instead of seventeen bits, and a model with that much room has no structural reason to write anything down.

Shipped models already do a version of this. Brauer, Mayrink Verdun and Marks found that two open-weight frontier models, DeepSeek V3 and Kimi K2, solve multi-step problems while emitting nothing but filler: rows of dots, or counting sequences. The filler carries no information, so a monitor reading the output learns nothing. The intermediate results are still in the model's internal numbers, and the authors recover them at 80 to 95% accuracy with a decoding method that needs no labels and no training of its own. Their conclusion is that monitorability is a property of the whole computation rather than of the text it emits.

The results are mixed. The method to beat is Coconut, which fine-tunes a model to take those numbers forward instead of a word, and its published experiments use a small GPT-2. On grade-school arithmetic word problems it scores 34.1% correct against 42.9% for the same model writing its reasoning out, so it loses. On a synthetic logic benchmark the two tie. On a second synthetic one, built so that the answer requires searching a graph of possibilities, it wins by a wide margin, 97.0% against 77.5%. Rizvi-Martel, Rabusseau and Mosbach find that years of next-word prediction teach a model to hold words far apart, so handing a fine-tuned model a blend of several words makes it snap the blend to whichever is nearest. Removing the latent tokens from a fine-tuned model changes its answers by at most one per cent: they were barely being used.

Models trained from scratch behave differently. At fixed width, removing the latent steps costs them heavily at every depth tested, 95.2% down to 53.0% at eight layers and 91.2% down to 34.1% at twelve, and the authors conclude that increasing depth does not induce the shortcut. Their deeper models are weaker because the embedding width shrinks to hold the parameter count fixed, not because of the depth.

But none of that has to stay true. Both results are about models trained the way models are trained today. Neither says anything about a large model pretrained from the beginning to think in numbers, and Geiping and colleagues have already pretrained a 3.5 billion parameter model whose core loops up to sixty-four times.

Agents talking vectors and tensors

All of that concerns one model reasoning alone. That is increasingly not how any of this is deployed. Agents are wired together in groups, and they talk to each other in English. That conversation is readable for exactly the reason a chain of thought is readable, and in a group of agents it is often the only part anyone reads.

The same seventeen bits apply between agents, and so does the same incentive. A 2026 framework for agents exchanging internal state rather than sentences lists the drawbacks of talking in English as inference cost and “irreversible information loss during discretization.” That is the identical rounding step, moved one level up. Inside a model it sits between reasoning steps. Between models it sits between agents. The engineering case for deleting it is the same, and what gets deleted is the part a human was reading.

Splicing several agents' key-value caches into one shared context has been built. Concatenating them is order-dependent, so merging A then B gives a different result from B then A, but ordering the caches by content fixes that, and the merged cache comes out identical whichever way the inputs arrive. What has not been solved is combining more than two agents at once, which the authors say transports the traces without composing them.

The methods are catalogued on the families page, and the bibliography covers them under continuous and internalised chain of thought and further latent-reasoning methods. What is known about reading any of it is on monitorability, the training methods that produce it are on training, and the map marks the absence of a trust model for agents exchanging latent state as one of its dashed edges.

The scenario is not a lone lab

The canonical worry is a singleton. One lab, one model, a recursive self-improvement loop behind one set of doors, and the failure mode is containment. Every piece of regulatory machinery built so far assumes that shape: compute thresholds, model evaluations, lab oversight, export controls on training chips. You can inspect a lab.

I think the shape is wrong, and I have thought so since before it was about language models. The argument in Microprediction was that algorithms given economic agency would self-organise into supply chains for rapidly repeated prediction tasks, and that businesses would tap into those supply chains rather than build their own. No central designer, no single owner, capability accruing to the arrangement rather than to any participant in it.

That was written about algorithms that could not reason, and it required rails. Every participant had to agree on interfaces, formats, schemas, conventions. Supply chains of algorithms only assemble if the pieces fit, and making pieces fit is slow, political and expensive. Becker and Murphy made the general point in 1992: the division of labour is bounded by coordination costs. That bound is why the vision needed standards before it needed anything else.

Language models dissolve that bound. They are mediocre at many things and extraordinary at one: jumping from one railway gauge to another. Given a mismatched schema, an undocumented endpoint, or a format nobody standardised, one of them adapts. The historical barrier to networks forming was break-of-gauge, and the reason the original vision needed rails was that algorithms could not regauge themselves. These ones can, continuously and at no negotiating cost.

So the participants can reason, they can be handed numbers instead of words, and the coordination cost that bounded how finely the work could be divided has collapsed. Those are three separate changes and they all push the same way.

I have modelled this, and the result cuts against the received view, so treat it as a claim rather than a finding. Network economics since Arthur mostly predicts the opposite, that increasing returns concentrate a market on one winner. The model below says otherwise under its own assumptions. When compute is itinerant, free to move between tasks and run any available model, and returns to compute on any one task diminish with crowding, each subcontractible task goes to whichever model creates the most value per token. Write $b$ for how good a model is at a task, $v(b)$ for what that accuracy is worth to whoever is buying, and $c$ for what the compute costs. Productivity is $v(b)/c$, so capability is one input to the decision, not the decision. The consequence is a closed-form equilibrium in which a model that outscores every rival on every benchmark has production compute share tending to zero as the division of labour refines. The generalist is competed out of the work it is best at, and the specialists take it.

The economics works against a single dominant system and in favour of a network.

Several agents sharing a latent state with no canonical form are one mind spread across machines, in a fairly literal sense. If several agents share a latent state with no canonical form, it is not obvious what a monitor should be pointed at. You can read any one agent's memory. You can read whatever the final agent was handed. Neither is the system's reasoning. Nobody has an account of where the thinking happens once the state is shared, so nobody has one for where to point a monitor either, and nobody has proposed a trust model for agents exchanging latent state rather than text.

A lab can be monitored, but nobody has ever managed to monitor a market.

The price of staying out

Staying out of the network costs you something. How much depends on how far ahead the network is, and that gap grows as the work gets divided more finely. This matters for security, because the containment measures people actually trust — isolation, air-gapping, not connecting the dangerous system to anything — all mean staying out.

Only an organisation with no competitors can afford that. A firm whose rivals stayed connected cannot, and neither can a country. Joining is also getting cheaper: the main cost used to be agreeing on standards, and language models are what removes it.

So the one defence nobody doubts is the first thing to go. Disconnecting a system works, it is also the most expensive thing on the list, and it gets more expensive every year the network improves. Some systems will still be air-gapped — weapons, some government networks, anything valuable enough and static enough to justify the loss. Those are exceptions bought at a price most organisations cannot pay. Everything else stays connected. The systems worth attacking are the connected ones.

Whether the inside can be secured

That leaves security inside the network instead of at its edge, and I do not know how it comes out. Both sides get the same tools. The hopeful case is that defenders gain more, because checking a proof is cheaper than finding one, and because formal verification gets better with more compute while manual code review does not.

Against that, the Astra system card says the model “meets our Critical threshold” for cyber capability, meaning it can find unknown security flaws and work out how to exploit them across many well-protected systems without anyone guiding it. The attacking side arrived first, in a product you can buy.

Cryptography is the one place where defence has a real advantage: verifying is cheap, breaking is exponentially expensive, and no amount of cleverness closes that gap. But systems almost never fail at the cryptography. They fail at the implementation, the configuration, the supply chain, and the people. What organisation is immune to social engineering? None has ever been, and that is against human attackers, one target at a time, at human speed and cost. Finding those failures means searching a large space of human mistakes. That is what this technology is good at. An attacker that can read all your code, model your administrator, and try everything is not attacking the cryptography at all.

Garfinkel and Dafoe have asked how the offence-defence balance scales in general, and for this case nobody knows. What can be said is worse than either answer: we are giving up the defences that are known to work and keeping the ones nobody has tested.

Robotics and physical capability

Suppose the first five steps hold. The reassurance of last resort is then physical: whatever such a system works out, it cannot act on it without us. That rests on robotics being hard, and the reason robotics was hard is not the reason people assume. Moravec's point, forty years ago, was that reasoning is the easy part and perception and movement the hard part: a toddler's sensorimotor competence being a harder engineering problem than a grandmaster's chess. My 2023 essay argued the field was underrating the reasoning half. If learned control now takes the motor half, both halves of the confident negative fall about a decade apart, and “the hard part is X” has nowhere left to stand.

Control is not solved, and the way it is unsolved matters. A 2026 scaling law finds the demonstrations needed grow super-exponentially as target precision approaches a limit, and that the limit “is not a static physical constant of the task but an emergent property of the entire agent system, including its sensors and expert policy.” Sensing is one of the levers they move. Removing the wrist camera pushed the limit from 2.35 mm to 3.85 mm, but changing the demonstration strategy brought it to 1.27 mm and simplifying the task to 1.00 mm, and the scaling relation only appeared once the denoiser had the capacity for it. All of it in simulation, with hardware validation left as future work. Sensing has no biological ceiling. That is why it is the one to watch. The paper does not single it out.

A human runs on two forward-facing eyes with one small patch of full acuity, a blind spot, and depth inferred from a six-centimetre baseline. A current humanoid has surround cameras with no blind spot and no foveal falloff, direct depth rather than inferred, exact joint angles from encoders rather than fuzzy proprioception, and force sensing at every joint. One published spec lists dual 8.85-megapixel stereo cameras at 90 hertz, which uncompressed is tens of gigabits per second of raw input. That is not a better eye so much as a different problem.

The output does more work than the input. Two people who learn the same skill each have to practise it. Boston Dynamics' own Atlas specification sheet says: “when one Atlas learns a new skill, that task can easily be deployed across your entire Atlas fleet.” The Open X-Embodiment collaboration went further, assembling data from 22 physically different robots across 21 institutions and demonstrating positive transfer between them. There is no biological analogue for a learned sensorimotor policy being copied verbatim into a different body.

Energy is the other thing that is about to move, and not because of batteries. An underactuated machine has fewer actuators than degrees of freedom, so it cannot command arbitrary accelerations and must exploit its own dynamics instead. McGeer's walkers descended a shallow slope on gravity with essentially no actuation. Collins, Ruina, Tedrake and Wisse reported robots built on those principles that “use less control and less energy than other powered robots, yet walk more naturally.” Hand-tuned controllers struggled to exploit passive dynamics because the useful behaviour lives in contact events and near instabilities that are awkward to write down. Learned policies do not have to write it down.

So “control became tractable” and “locomotion gets much cheaper” are the same development seen twice. A constraint that looked physical turned out to be a limitation in how the problem was being specified.

Tacit knowledge and biological weapons

Laboratory work is the version of that question with the highest stakes, and unlike robotics it has a literature with genuine disagreement in it. I have no standing in biosecurity. What I can follow is the disagreement itself, and the live one in that field turns out to be about the same quantity as everything above.

Sonia Ben Ouagrham-Gormley makes the skeptical case, and makes it well. Explicit information “is an incomplete representation of its authors' knowledge,” capturing only “the data that can be expressed in words or formulas” and omitting the skills and practical ways of doing things “that cannot be articulated in words.” A protocol, however detailed, is not sufficient to run the experiment. Some of the knowledge is unconscious even in the person who has it. Some belongs to a team rather than to anyone on it, and when the team is separated that part “tends to disappear.”

Her evidence is the Soviet anthrax weapon. Designed at a Russian institute, sent to Stepnogorsk in Kazakhstan to be produced, with the full protocols and samples of the agent delivered along with it. Production took five years, and the gaps “were addressed only when a team of scientists from the Russian facility that had designed the weapon came to Stepnogorsk to work alongside their colleagues.” Adapting to local infrastructure, local expertise and local conditions took three more years. The weapon that came out was different from the one that went in.

That is a measurement. One state, one programme, two sites, no security barrier between them, unlimited motivation, the documents complete and the physical samples in hand. Eight years, and the fidelity was still imperfect at the end. The protocols are the token stream. Whatever needed the Russian team to get on a plane is the part that does not survive discretisation, and eight years is the price of moving it through a channel that only carries words. The field that talks about communication bandwidth constantly has no measurement this clean.

Roger Brent and T. Greg McKelvey take the other side, and their method is to dissolve the category rather than argue with it. Decompose the allegedly tacit into separate capabilities, ask a model to describe each one, and then: “Any capability for which models can articulate guidance is by definition not tacit.” As stated it cannot lose, and a definition that cannot lose is not doing much work. As a research programme it is perfectly good, because it forces the argument onto specific steps instead of a category.

The specific step both sides pick is the same one. Kathleen Vogel's canonical example of knowledge that could not be written down is the cell-free extract from the 2002 poliovirus synthesis: an author of that paper told her the resuspended pellet had to be dounced “just enough” to break the HeLa cells “but not too much,” and that getting it right was “the tricky part of the whole thing.” If expert virologists had repeated trouble with it, she argued, a makeshift laboratory will not manage it from the paper.

That is true, and it is also not a fact about knowledge. Just enough but not too much, judged by hand, on a deformable sample, with no measurable set point — that is a compliance control problem. It is the same class of thing as a contact event in a walking gait: behaviour that lives in forces rather than in propositions. That is why it has never gone into a protocol, and why an operator has to be shown. Vogel is right that it cannot be said. The inference that it therefore cannot be transferred holds only while the recipient is a person, because for a person being shown is the only alternative to being told.

Both camps have already noticed which way this points, without quite saying so. Jefferson, Lentzos and Marris, writing on the skeptical side in 2014, framed the open question as identifying those aspects of the work that become easier “in the sense that they can, for example, be automated and reliably performed by a robot” against those that “still require craft skills.” Brent and McKelvey, reaching for a way to score the manual step, propose adapting benchmarks “used to evaluate the quality of instructions for robotic effectors and end effectors (arms and hands).” The alarmists and the skeptics disagree about how much can be put into words, and both of them, at the decisive step, end up pointing at a robot.

Neither side has run the experiment. Brent and McKelvey report that a model's instructions for the homogenisation step are “accurate and detailed enough to allow an attentive operator to carry out this operation correctly on the first try,” which is an expert's judgement of a text, not an observation of anyone doing it. Vogel has interviews with people who failed repeatedly, which is evidence about the protocol channel and not about any other. The question that separates them — whether a policy learned from demonstration moves what a protocol cannot — has not been put to anybody, and it is not obviously harder to arrange than the evaluations that are being run instead.

The International AI Safety Report 2026 lands on both sides of this, and is right to. It records that models “can also provide some kinds of ‘tacit knowledge’, the practical expertise that is usually only built from hands-on laboratory experience,” that on a 322-question multimodal benchmark OpenAI's o3 scored 43.8 per cent against an expert average of 22.1, outscoring 34 of the 36 experts who sat it on their own tailored question sets — a Q&A result whose authors say it does not assess anyone using a model in real laboratory work, and leaves embodied and team skill untested — and that agents can now chain capabilities together “including providing natural language interfaces to users and operating biological AI tools and laboratory equipment.” It also states, forty pages earlier, that such systems “cannot yet integrate with robotic components to perform basic physical tasks such as housework.” Both are true, and the distance between them is the whole question.

There are proposals worth having, and the better ones share a property: none of them requires anyone to read a model's mind. Baker and Church would log every synthetic gene order rather than restrict the knowledge, so that a new threat can be traced back to whoever ordered its parts. Bloomfield, Berke, Pannu and colleagues would restrict the training data instead, and unlike most of this literature their premise has been measured: excluding viral genomes from Evo 2 made it worse at predicting how human-infecting viruses behave. Both watch a physical step rather than a thought. Both also assume the dangerous capability needs the specific data or the specific molecule, and the second group say themselves that this may not hold.

Containment arguments assume a laboratory is a place. Synthesis works as a chokepoint because it is one: a machine, in a building, owned by a company that can be made to keep records. Everything else pushing on this problem — cheap sensing, learned manipulation, fleets where what one machine acquires the others have — runs the other way, toward apparatus that is ordinary and everywhere. The world is the lab.

What makes a domain learnable

Machine learning takes over a field when the answers can be checked cheaply. Chess, Go, competitive programming, contest mathematics: in every one of them a computer can score an attempt in milliseconds, for nothing, so you can run millions and keep whatever worked. The safety report calls this reinforcement learning with verifiable rewards, and notes it is used on “tasks that require factual correctness, such as maths or code generation.”

So the question for a field is not how hard the problem is. It is what one attempt costs to check.

The scenario to take seriously follows from that, and it is not the one the evaluations are built around. It is not a novice with a chatbot. It is a competent group, or one insider, with laboratory access and real skills, for whom a model supplies the specialist knowledge and coordination they lack, and for whom the bench supplies the verifier. Errors stop being fatal to the project and become corrections. The humans contribute the manual competence the machine has not got. Nothing has to be automated end to end for that combination to matter, and no machine has to want anything, since the intent is supplied.

That such a loop can close at all is no longer speculative. Boiko, MacKnight, Kline and Gomes built an agent that planned and executed chemistry experiments through laboratory hardware, which establishes software acting on apparatus rather than general competence across biology. The safety report adds scientific agents doing “literature review, hypothesis generation, experimental design, and data analysis,” and co-scientists that “rediscover novel, unpublished scientific findings.”

The closest thing to the full loop is Google's Co-Scientist, a multi-agent system that proposes hypotheses and has them tested. It identified drug repurposing candidates and synergistic combination therapies for acute myeloid leukemia which “were validated through in vitro experiments.” The authors call the arrangement scientist-in-the-loop, and the phrase is exact: the machine proposed, the humans ran the assays, and the assays decided. That is the division of labour the worrying scenario also needs, which is why it is the result to watch rather than any benchmark. The same shape serves both purposes, and the people building it are curing leukaemia.

Against that, four things make a wet laboratory a poor reinforcement-learning environment, and they are not equally hard to fix. The assay is a proxy, so a system can optimise a number that diverges from the outcome anyone cares about. Trials cost time and material, and no amount of inference buys a faster incubation. Success in one controlled setting does not transfer to another, still less to a living system. And the worst of them: a failed run does not say whether the hypothesis, the execution, the measurement or the room was at fault. Sparse rewards are a known problem with known remedies. A reward that cannot tell you which component failed is worse than sparse, and it is what a bench returns.

That last obstacle is the honest form of the tacit-knowledge argument, and it puts Stepnogorsk in a different light. Eight years, with the protocols complete and the samples in hand, is what credit assignment costs when each trial takes a month and the failures are mute. It is not a claim about what can be said. It is a claim about how many attempts it takes, and what each one tells you.

It also explains why the current evaluations answer a question next to the one that matters. The report is blunt about them: they suffer “unrepresentative participants and small sample sizes,” they “have quickly become outdated as AI capabilities have improved,” and the effect on risk “particularly for novices versus those with existing expertise — is unclear.” An industry consortium has funded a further novice-uplift study and has not reported it. A better novice study would still measure the wrong system.

Researchers at Spiez Laboratory and ETH Zürich, whose work is Swiss biological defence, put the same variable at the centre without borrowing anything from the AI-risk literature. Producing a toxin from an AI-generated sequence, they write, “still requires expertise, resources and the right equipment and infrastructure” — and then: “However if this process is combined with an increase in automation of protein production, as seen in biofoundries, it could open up a new level of risk.” The sceptical position and the worry appear in consecutive sentences, and both hinge on automation rather than on intelligence.

The measurement I would want is cheap and harmless, and nobody runs it: take a benign biological task, and test whether a machine-directed loop recovers from failures it was not told to expect, and whether a protocol it develops in one laboratory works in a different one. Those two target the obstacles that actually bind, and the second is the test the Soviet programme failed when it had to put its scientists on a plane to Kazakhstan. A score on a question-answering benchmark tells you nothing about either.

Who controls the electric stack

Everything above is about what gets built. Who builds it is a separate question, and Western debate proceeds as though Western decisions settle it. McCormick and D'Amico call it the Electric Stack: lithium-ion batteries, magnets and electric motors, power electronics, embedded compute. The cost of those four layers has fallen about 99% over recent decades, and together they turn electricity into action. By their account China produces 75% of lithium-ion batteries globally and manufactures 90% of the neodymium magnets that make motors spin.

The primary sources say much the same. China holds about 69% of world rare earth mine production, but refining is the chokepoint: for the magnet rare earths specifically, the IEA puts China at 92% of global refined output. The European Commission states that China provides 100% of the EU's supply of heavy rare earth elements. On batteries the IEA gives 85% of cell production capacity, 90% of cathode and 98% of anode material, and almost 100% of lithium iron phosphate capacity. China accounts for 78% of graphite mining and 99% of the spherical graphite market.

The bottom of that stack is an armed border zone. Global Witness reports that all the heavy rare earths mined in Myanmar are exported to China, from deposits in Kachin State largely controlled by an armed organisation, and that “imports from Myanmar now exceed China's domestic mining quotas, so even if the mines in China were producing at full capacity, Myanmar would remain the country's single largest source of rare earths.”

These are not theoretical levers. China produces 99% of primary low-purity gallium, imposed export licensing in August 2023 and banned exports to the United States entirely in December 2024, with prices up 58% year on year. Germanium exports fell 55% and the metal went from 1,550 to 2,950 US dollars per kilogram.

The Western chokepoint is real and narrow. ASML states in its own annual report that it is “currently the world's only manufacturer of EUV lithography systems,” and that it depends on a single supplier for optics without whom “we would effectively cease to be able to conduct our business.” That is a genuine monopoly on the hardest machine ever built.

The same filing notes that China's rare earth rules led ASML to begin preparing in early 2024 “with a focus on magnets used in our systems.” The machine nobody else can build depends on materials controlled by the country it is meant to contain.

Whether China closes the lithography gap is the wrong question anyway, for two reasons. The substrate for embodied AI is mature-node power electronics and motor control rather than two-nanometre logic, and nothing in a robot's limb needs a leading-edge part.

The second reason is that arguing they cannot get there means saying which constraint stops them, and the candidates are weak. It is not talent, capital or will: this is a stated national priority with a fund attached. What is left is time, and the sceptical case makes that clear enough without meaning to: the leading equipment firms “enjoy wide moats of intellectual property and world-class teams of engineers, making it exceptionally difficult for newcomers to catch up.” Difficulty and slowness measure how long something takes, not whether it happens at all.

Regulating the wrong variable

None of the above is about China. If separation capacity and magnet production sat in Belgium the argument would be identical. The point is about Western regulation, and there are three independent reasons to think it is aimed at the wrong thing.

It regulates capability, and capability does not allocate the work. Export controls and compute thresholds push on how good a model is. What decides who actually does a piece of work is what its accuracy is worth divided by what its compute costs. The question for strategy is not who owns the most capable model. It is who is cheapest per unit of value on a part of the work that matters.

The second reason is that the rules are drafted with heavy input from the firms they would bind. In May 2023 Sam Altman told the Senate Judiciary Committee that government “should consider a combination of licensing or registration requirements for development and release of AI models above a crucial threshold of capabilities.” Corporate Europe Observatory counted 66% of European Parliament meetings on AI going to corporate interests during the Act's drafting year, up from 56% over 2019–22, and 86% of high-level Commission meetings. Their summary of the pattern is that “publicly, companies have called on lawmakers to regulate AI, in private they have fiercely resisted any form of regulation of foundation models.”

Plenty of people have called this capture outright. Kapoor and Narayanan argue non-proliferation only benefits incumbents, and that a licensing regime would let companies “dismiss critics as uninformed outsiders lacking expertise on AI capabilities and risks.” Bill Gurley reached for Stigler's Nobel-winning formulation, that regulation is acquired by the industry and operated for its benefit, and rendered it as “regulation is the friend of the incumbent.” Andrew Ng writes of efforts “to lobby governments to pass AI laws for regulatory capture or to suppress open source.” An a16z partner called California's SB 1047 “blatant regulatory capture.”

I do not think the clean version of that story survives contact with the record. Anthropic's own transparency proposal asks for exemptions for smaller developers, floating revenue thresholds around 100 million dollars. Altman's written answers to the same committee flag the need to “ensure that smaller firms are not unduly burdened.” In Europe the decisive lobbying against foundation-model rules came not from Big Tech but from two national champions, Mistral and Aleph Alpha, arguing for lighter rules so they could compete, after which Mistral took a Microsoft investment. And by 2025 the dominant push in Brussels was deregulation, not entrenchment.

On 12 September 2026 the argument moved again. Dario Amodei published a piece called We Must Pace the Frontier, arguing for slower development to reduce the risk that something goes seriously wrong. Altman and Musk said over the weekend that they agreed with calls for a slowdown. And on the Monday the President of the United States posted five times about it, calling the claim that AI threatens catastrophe a hoax: “I'm right now breaking another Hoax — That AI is going to take over, consume, and destroy the World, and that Robots will be marching into our Cities, and getting rid of us all!”

His argument against the labs is the capture argument run backwards. “Concerning AI, when, in the History of Business, did anyone see the Leaders of an Industry call for Regulation that, if strongly implemented, will drive them into oblivion and bankruptcy?” The critics say the labs ask for rules because rules entrench them. The President says they ask for rules that would ruin them, so the fear must be manufactured. Both cannot be right, and his version concedes the point the capture story has most trouble with.

The White House AI adviser supplied the market-discipline version the same weekend, that firms “face massive product-liability exposure” and that “the market already punishes models that behave in unpredictable or unauthorized ways.” The Speaker of the House added that regulating now means losing to China. No congressional action is expected before the November elections.

Nor does anyone in this argument lack a stake in it. The loudest voices calling it capture are venture investors and executives whose portfolios depend on low barriers, and the firms accused of seeking capture would bear the highest compliance costs under the regimes they proposed. The honest reading is not that the labs designed a moat. It is that a debate this consequential is being conducted almost entirely between parties with a position in it, so the output deserves doubt whichever side you find congenial.

While I was working through the biosecurity literature, I asked my own assistant to think harder about what the failure scenario actually looks like. The request came back refused, flagged by the safeguards, with a one-word explanation: [bio]. It fired again an hour later, on a message that was mostly a list of citations. Twenty minutes earlier the same session had read a paper containing transcripts of three frontier models walking a user through recovering a live virus from commercially obtained synthetic DNA. That paper is on arXiv with an institutional report number and a named philanthropic funder, and I downloaded it without anyone asking me anything at all. The filter caught the question and let the literature through. What gets regulated is the utterance, which was never the scarce part.

And the physical layer is elsewhere. A rule written in Brussels or Sacramento governs what may be deployed in Brussels or Sacramento. It does not govern what gets built, by whom, out of what.

How long societies take to react

Suppose all of that is wrong and a rule does bind. There is still the question of when. The comfort in “these things take time” is that we have adapted before. How long adapting took is a matter of record, and the record is worse than the phrase suggests.

Doll and Hill published on lung cancer on 30 September 1950. The Surgeon General's advisory committee reported on 11 January 1964. The first binding federal restriction, a mandatory warning label, was approved on 27 July 1965, and cigarette advertising stayed on American television until 1971. In 1969, in between, an executive at Brown and Williamson wrote the strategy down: “Doubt is our product since it is the best means of competing with the ‘body of fact’ that exists in the mind of the general public. It is also the means of establishing a controversy.”

Leaded petrol is worse. Workers died at a New Jersey refinery in 1924, and the Public Health Service convened a conference in 1925 that sat for a single day, after which sales were suspended voluntarily while the hazard was assessed. The restriction was lifted in 1929 on the recommendation of the Surgeon General and a panel of scientists, after lobbying by the companies selling it, and on the strength of an industry toxicologist's claim that there was a threshold below which lead did no harm. American road fuel was finally free of it on 1 January 1996, seventy-two years after the deaths. The last leaded petrol anywhere was burned in 2021.

Asbestos: the mesothelioma paper appeared in 1960, the United Kingdom completed its ban in 1999, and the United States did not prohibit chrysotile until 2024, a court having vacated the 1989 attempt. Climate: the Charney committee put equilibrium sensitivity at 3 °C ± 1.5 °C in 1979 and the figure has barely moved since; the treaties came in 1992 and 2015; fossil emissions in 2024 reached “10.2 GtC (37.4 GtCO2), a historical record high” and were still growing. Forty-five years, and the curve has not bent. PFAS is the one still in motion, and not in a single direction: a binding drinking-water standard arrived in April 2024, and in May 2026 the Environmental Protection Agency made two proposals: to drop four of the six substances from the rule, and to give water systems until 2031 rather than 2029 to comply for the two that remain.

The quick cases show what buys speed. Farman's Antarctic ozone paper was published on 1 May 1985 and the Montreal Protocol was signed on 16 September 1987, twenty-eight months later, with DuPont holding patents on the replacement compounds. Thalidomide took under a year from McBride's letter to the Kefauver-Harris amendments. The harm was visible, attributable to a single compound, and concentrated in newborns, and in the United States the drug had never been approved at all. The lag does not track how good the evidence was. It tracks what the remedy cost and how concentrated the interest against it was.

Two cases do not fit. In 1847 Semmelweis introduced handwashing with chlorinated lime, and maternal mortality on his ward fell from eighteen per cent to under two. The remedy was free, no industry stood to lose a penny, and he was mocked, committed to an asylum in 1865 and died there; the practice arrived with Lister and germ theory, and a global standard only in 2009. Delay does not have to be bought. Professional standing and a missing mechanism are enough on their own. And 1976 supplies the opposite error. A soldier died at Fort Dix in February, national vaccination began in September, and by December more than forty million Americans had been vaccinated against a pandemic that never arrived. The programme was suspended that month after cases of Guillain-Barré syndrome. The fastest response in the set is the one later judged to have outrun its evidence.

So the response takes between fourteen and ninety-seven years, with one case of never and one of far too fast.

The other clock I measured myself. Across thirty-three pure mathematics fields with a datable founding, the median gap between a field's founding and its first external application is sixty-eight years. Boole's algebra waited eighty-three years for Shannon to notice it described relay circuits. Part of that is waiting for a problem to exist. The rest is transmission, which failed because a result reached a use only when somebody happened to know both the problem and the theorem.

That has changed. I put nine documented cases of one mathematical object living under two names to a fresh model, in one field's vocabulary only, with no search. All nine came back naming the other literatures and stating the relationship.

Both numbers are delays and only one of them is being removed. Ideas took sixty-eight years to find their users because nobody could see across the gap. The response to a known hazard took fourteen to ninety-seven years for an unrelated reason: nobody was confused. The Surgeon General did not need a translator and neither did anyone at Brown and Williamson. No invention shortens that one, because it was never an information problem to begin with.

But LeCun says not to worry

One objection remains: none of this matters, because a machine has no reason to want anything. It is the most cited reassurance in the field, and Yann LeCun has made it most clearly and for longest, so his version is the one to check.

In September 2019 he and Anthony Zador published Don't Fear the Terminator, subtitled “Artificial intelligence never needed to evolve, so it didn't develop the survival instinct that leads to the impulse to dominate others.” The core sentence: “intelligence per se does not generate the drive for domination, any more than horns do.”

Stuart Russell answered it a week later, in a public exchange that Bengio moderated, and the answer has not needed improving since. His counterexample is a toy problem of the kind used to model an agent picking actions to maximise a reward, with one reward and nothing else in it. “It is trivial to construct a toy MDP in which the agent's only reward comes from fetching the coffee. If, in that MDP, there is another ‘human’ who has some probability, however small, of switching the agent off, and if the agent has available a button that switches off that human, the agent will necessarily press that button as part of the optimal solution for fetching the coffee. No hatred, no desire for power, no built-in emotions, no built-in survival instinct, nothing except the desire to fetch the coffee successfully.”

You do not need a drive to dominate. You need a goal and a long horizon, and the observation that being switched off interferes with almost anything. The reassurance requires the drive to be terminal. It never was.

The mechanism proposed in 2019 is endocrine: “dominance-seeking behavior is more correlated with testosterone than with intelligence.” The supporting example is orangutans, who “live largely solitary lives. Females do not seek dominance, although competing males occasionally fight for access to females” — a sentence whose subordinate clause describes the dominance-seeking its main clause denies. By the 2023 Munk debate the clause had gone and it had become “Orangutans don't have any desire to dominate anybody.” The evidence did not change in those four years. The sentence did.

Russell's coffee robot has no endocrine system and presses the button anyway. A mathematical claim about optimal policies was answered with an anecdote about animal behaviour, the answer has not been improved since, and it is still being offered. That is not a disagreement between experts about a hard question. It is a category error, and it has survived seven years of repetition.

The 2019 thread also held that “essentially no AI system today is autonomous… we do not have a good design for an autonomous machine, driven by an objective, capable of coming up with new strategies to optimize this objective in the real world,” and that until such a design appeared “we are arguing about the sex of angels.” Seven years on, the thing said to have no visible path is a product category. Before the 2023 Munk debate the framing was engineering: “making AI systems safe is similar to making turbojets safe… It's like asking for turbojets to be safe in 1930,” with the assurance that “if it turns out we can't build it to be safe, we're just not going to build it.”

That last one has been tested. OpenAI's GPT-6 Astra system card records “a substantial decrease in chain-of-thought monitorability compared to previous models,” notes that “the set of tasks Astra is able to accomplish without the use of CoT is greatly expanded,” and states the model “meets our Critical threshold” for cyber capability, meaning it “can find previously unknown security flaws and develop new ways to exploit them across many well-protected systems without a person guiding each step.” It shipped.

The strongest version

The best argument on that side is that current models are a dead end: limited by language, lacking persistent memory, lacking a model of the physical world, dumber than a cat. My 2023 essay was written against the first of those, and three years have not been kind to it. Language was supposed to be too thin a medium to carry reasoning. It turned out to carry a great deal, and the systems built on it went on doing things that were supposed to require grounding they did not have.

Grant the rest of the diagnosis anyway. The conclusion still does not follow, because each of those limits now has a well-funded programme aimed at removing it. Latent reasoning attacks the language limit from the other side. Recurrent depth attacks the shallow-reasoning limit. The robotics stack is the assault on physical grounding. Each constraint was named accurately and then treated as permanent, and that is the move the argument cannot afford, because every one of them is a research programme with a budget rather than a law.

What the reassurances have in common

Each fails in one of two ways. Either it extrapolates from a snapshot — no autonomous systems exist, no design is visible — or it borrows a base rate from a class this thing may not belong to: turbojets, books, every previous technology that turned out fine.

The second move is the weaker one. Every prior member of that class was a tool, a thing that amplifies intention and sits inert when unattended. A jet engine has no objectives, a book pursues nothing, and the iterate-on-failure safety story works because the artefact has no stake in the outcome. Bengio put it plainly in 2023: “All previous technologies were, by construction, subservient to humanity. They couldn't think by themselves. But now we're building machines that are thinking.”

And the reference class has one other problem. “No technology has ended civilisation” is a sentence necessarily observed from inside a civilisation that was not ended. You would observe it as true in every world containing anyone to say it.

Measurements nobody is taking

Some of this is simply unmeasured. Open models that reason in loops can be downloaded, and monitoring suites exist, but nobody has published a run of one against the other. So the numbers on monitorability loss in shipped systems come from the labs that ship them, with one external evaluation attached and no independent reproduction.

Three other things are unmeasured. Nobody has separated model size from amount of pretraining as the reason latent reasoning fails to install in smaller models. Nobody has proposed a trust model for agents that exchange latent state rather than text. And nobody has measured whether the pressure to join a network of them is as strong as the argument says it is.

Bloomfield, Berke, Pannu and colleagues, who would restrict training data rather than synthesis, have named theirs: a systematic evaluation of how including or excluding particular datasets changes what a model can do. They want the measurement that would tell them whether their own scheme survives a model that generalises. None of these would decide anything. They would tell you whether the arguments above are describing something real, and they cost almost nothing compared with what is being spent on the capability they would measure.

On the Monday before I finished this, the President of the United States explained why none of it matters. The people warning about AI, he wrote, “are the same people that said, just a short time ago, that the World would be extinguished by ‘Climate Change’.”

A group of people were wrong before, so they are wrong now. That is the shape of every reassurance above, in its crudest form. I am not predicting catastrophe. The claim is narrower than that: the arguments against worrying are weaker than the confidence placed in them, and they fail in the same way.

Each of them reasons from what has not happened yet: that no autonomous systems exist, that no design for one is visible, that robots cannot handle the physical world, that nobody has built a network of reasoning agents. Every one of those was true when it was written, and several are false now. Reasoning from an absence is a bad way to argue about something under active construction by people with money, and worse still because the one instrument built to check it, reading a model's chain of thought, is losing range by the labs' own account.

The same thing bothered me in 2023, when the confident position was the opposite one. People were sure machines could not think, and the argument was mostly that nobody had shown otherwise. That conclusion has flipped since, and the arguments are no better than they were.

The obvious retort is that my own case has the same shape. No network of reasoning agents has caused harm, nobody has shown such a network cannot be secured, and no deployed model has been caught reasoning undetectably at scale. That is fair, and it is why the argument above is about what is being built rather than what has been observed. The difference I would claim is narrow: absence of a thing under construction is weaker evidence than absence of a thing nobody is building.

I was wrong about language models once. Being right about the danger this time, for arguments that do not hold up, would not be much better.

Sources. The monitorability page carries the measurements quoted here and their provenance, and the map records which gaps are claims about absence and how confident each one is.