latent reasoning

← Home

The decoding loop

One line of the decoding loop, deleted.

The decoding loop

Take a decoder-only language model, and let $f$ be the forward pass that maps a sequence of input embeddings to the final-layer hidden state at the last position, $W$ the unembedding matrix, and $E$ the embedding table. Autoregressive decoding is then the loop

$$h_t = f(e_1, \dots, e_t), \qquad x_t \sim \mathrm{softmax}(W h_t), \qquad e_{t+1} = E[x_t].$$

The middle step is a discretisation. The composite map $q = E \circ \mathrm{sample} \circ \mathrm{softmax} \circ W$ sends $\mathbb{R}^d$ to $\mathbb{R}^d$, but its image is the finite set of $|V|$ rows of the embedding table. Two things it is not: it is not nearest-neighbour rounding in embedding space, it is stochastic selection from a distribution, followed by a table lookup, and $\log_2|V| \approx 17$ bits is an alphabet capacity bound, $H(x_t) \leq \log_2|V|$, not the realised information content of a step. Whatever the model computed at step $t$ that is not recoverable from $x_t$ does not travel along this path to step $t+1$.

Latent reasoning is what happens when you delete $q$:

$$e_{t+1} = h_t.$$

That is the entire mechanical content of the continuous-thought family. The model keeps running its own forward pass on its own output, as before, but the output is now the unrounded state. Everything else, the architecture, the attention, the KV cache, is untouched. What changes is the width of the channel between one step of reasoning and the next.

Replayability, resources, monitorability

A cached decoder does not start from nothing at each step. If $C_t$ is the key-value cache, the update is closer to

$$(C_{t+1}, h_{t+1}) = F_\theta\big(C_t,\; E[x_t]\big).$$

So there are two paths from one step to the next, and they behave differently. The feedback path runs from the top layer back to the next input embedding, and that is the one the discretisation narrows. The cached-activation path carries retained intermediate activations at every layer and position, and attention at later positions reads them directly. Calling the emitted token a seventeen-bit bottleneck on all persistent computation is therefore wrong.

This is exactly why filler tokens work. A row of identical dots carries no task-specific token identity, yet each new position still attends over the whole retained cache and adds computation. What bounds that computation is the decoder's layer-and-position dependency graph, not the size of the vocabulary. Contentless padding buys parallel workspace at fixed depth. It does not buy unbounded serial depth.

Three claims worth keeping apart

Replayability. Given the model, the full prompt, a fixed execution schedule and deterministic forward operations, the cache can be recomputed from the token history, so the run can be reproduced. But a deterministic latent recurrence $z_k = F_\theta^{(k)}(z_0)$ satisfies the same property: its state is fixed by the input and the iteration count. Replayability does not distinguish text reasoning from deterministic latent reasoning.

Computational resources

How much serial depth and parallel width a scheme makes available is an architectural question, treated on the theory page. It is where the real differences between token-space and latent schemes live.

Monitorability. Whether a human or an automated monitor can read what the model is doing. This does not follow from either of the above. Determinacy is not legibility: a transcript can fix a computation completely while telling you nothing you can read off it, which is precisely what filler tokens demonstrate: recovering that hidden computation took an unsupervised decoder operating on activations, not a reader. Monitorability is an empirical property of how models are currently trained, not a structural consequence of discretisation. Monitorability has the measurements.

What deleting the discretisation does change is narrower than the older framing here claimed: it widens the feedback interface, so intermediate state need no longer be funnelled through a token vocabulary on its way to the next step. That is a real change and worth studying. It is not the removal of the only channel, and it is not by itself the removal of oversight.

Two axes

Methods in this literature vary along two independent axes, and most confusion about the field comes from conflating them.

Position of the loop

A reasoning step can be a new sequence position, the loop runs around the outside of the forward pass, one token of context consumed per step, or it can be a new pass through the same layers, the loop running inside the stack with no sequence position spent. The first buys serial computation at the cost of context length. The second buys serial depth at the cost of latency, and is where recurrent-depth architectures live.

Content of the channel

A token, a continuous vector, or nothing at all. The third case is not a joke: filler and pause tokens add sequence positions and compute while passing essentially no task content through the channel, which makes them the cleanest experiment anyone has run on what intermediate tokens are actually for.

WHAT SURVIVES THE LOOP WHERE THE LOOP CLOSES around the forward pass costs a sequence position inside the layer stack costs a pass through the weights a token ~17 bits a vector ~10⁴ bits of width no token content activations still flow Chain of thought the transcript is a sufficient statistic Continuous thoughts feed hₜ back as eₜ₊₁ Filler & pause tokens parallel workspace, fixed depth degenerate: a loop that emits a token has left the stack Recurrent depth iterate a block to convergence Adaptive depth halt when a learned signal says so

The two shaded cells are what the field means by latent reasoning. The literature on the unshaded cells is what tells us whether it should work.

Arguments for latent reasoning

Three arguments recur, and they are not the same argument.

Bandwidth. A reasoning step that must be rounded to a token cannot carry a probability distribution, a partial ordering, or a superposition of candidate answers to the next step. It must commit. A continuous state can carry all of them, so search that would require backtracking in token space can in principle proceed in parallel. This is the argument that continuous thoughts encode a frontier rather than a path, and it is the one with the most direct empirical support.

Expressivity. Serial steps add computational power to a transformer in a way that width and parameters do not. The theory here is about token-space steps and is genuinely settled in outline. Whether continuous steps add strictly more is a separate and much less settled question. The theory page has the bounds.

Cost. Natural language is a wasteful encoding of intermediate arithmetic. If the same computation can be done in fewer, denser steps, inference gets cheaper. This is the argument that motivates most of the compression work, and it is the one where claimed gains are most sensitive to how the baseline was tuned.

Next: the families of methods organised by the two axes above, what the theory actually proves, and the monitorability question that the bandwidth argument raises in its sharpest form.