Training

Supervised and reinforcement recipes for latent reasoning, and what each one requires.

Two routes exist. Supervised fine-tuning teaches the model where a latent step should go by showing it an explicit chain of thought and removing the tokens. Reinforcement learning scores the answer and lets the model find its own steps. The first route inherits the teacher's trace. The second has to deal with a property of the reference architecture that the supervised route does not: a deterministic latent rollout has no policy to optimize. The reinforcement learning methods differ mainly in where they inject the noise that creates one. Most of the methods that use reinforcement learning apply it on top of a supervised warm-up rather than in place of one.

Distillation and the curriculum

The supervised recipes all start from a model that can already write the chain of thought, and differ in what replaces the chain once it is removed. Stepwise internalization (Deng, Choi & Shieber 2024) replaces it with nothing. Tokens are deleted from the front of the rationale on a schedule, a fixed number per epoch, with the optimizer state reset at each deletion, until the model goes from question to answer with no intermediate positions at all. The evidence is answer accuracy. Whether the hidden states now reproduce the deleted steps or implement some other route to the same answers is not tested: the paper runs no probe or intervention and lists probing as future work. The procedure is also fragile. Of two 9×9 multiplication runs differing only in seed, one solved the task after the last chain token was gone and the other did not. Without the smoothing that randomizes the deletion point, accuracy drops to zero and does not recover. Without the optimizer reset, the same thing happens within about a hundred steps.

Coconut (Hao et al. 2024) keeps the positions and changes what fills them. The schedule is the same, but at stage $k$ the first $k$ reasoning steps of the written chain are replaced by $k \times c$ continuous thoughts, each the last hidden state of the previous pass fed back as an input embedding. The chain still occupies sequence positions and still runs left to right. It no longer passes through the vocabulary. The loss is masked on the latent positions, and the optimizer is reset between stages. The curriculum is necessary. Trained directly at the final stage, with no intermediate stages, Coconut falls from 34.1% to 14.4% on GSM8K, from 99.8% to 52.4% on ProntoQA, and from 97.0% to 76.1% on ProsQA, below the no-chain-of-thought baseline on all three. The authors' summary is that the model "still needs guidance to learn latent reasoning." Three costs come with that guidance: a ground-truth chain of thought for every training example, a multi-stage schedule, and $n+1$ sequential forward passes to train $n$ latent thoughts, because each thought is the previous pass's output.

CODI (Shen et al. 2025) removes the schedule. The same model runs an explicit-chain task as teacher and a latent-chain task as student, and an L1 loss pulls the student's hidden activations, at every layer, toward the teacher's at a single designated token, for example the colon in "The answer is:". Training happens in one stage, and the paper attributes Coconut's shortfall to forgetting across curriculum stages. It is the first latent method to match explicit chain of thought on GSM8K at GPT-2 scale, at a 3.1$\times$ compression rate. CODI still needs the written chain for every example, because the teacher task is trained on it.

Whether the curriculum can be dispensed in general is contested. Zou, Xiong & Liu (2026) model non-curriculum training as imitation of the model's own latent states, and show that if that distribution is already worse than the expert by a fixed gap, more samples from it cannot close the gap. The distributional bias is the theorem's hypothesis rather than its conclusion, so what is established is a failure mode conditional on the mismatch, not that direct training must produce one. Coconut's own ablation is the direct evidence that the curriculum mattered in its setup: 14.4% against 34.1% on GSM8K, 52.4% against 99.8% on ProntoQA, 76.1% against 97.0% on ProsQA. The argument covers training in which the only supervision is the final answer, so the latent states are shaped by the model's own trajectories. CODI's distillation loss supervises the hidden state directly against the teacher's activations, so it falls outside that setting. Both agree that a final-answer loss alone does not determine what the latent states should contain.

What outcome supervision loses

On GSM8K-Aug with GPT-2, Chen et al. (2026) find outcome-only latent training lands between 9.8% and 18.3%, against 18.7% for answering with no chain at all. For that run the authors report a gradient norm concentrated on the first latent position, with positions two through six receiving little, and a PCA plot in which the latent states move away from the explicit chain's embeddings. They read these as the cause of the failure, but neither is tested by intervention, and a gradient norm depends on the scale of the coordinates it is measured in. Their Llama-3.2-1B and ProntoQA runs reproduce the accuracy gap, not the diagnostics.

A progressive curriculum that replaces written steps with latents one stage at a time lifts accuracy to 31.2%. The curriculum also resets the optimizer at each stage, and without the reset it reaches 24.7%. Supervising each latent by asking a decoder to reconstruct the corresponding written step lifts it to 39.1%, against 43.1% for explicit chain of thought. SIM-CoT (Wei et al. 2025) is the same fix applied to Coconut and CODI: an auxiliary decoder, discarded at inference, is trained to regenerate step $k$'s text from latent $k$ alone, which prevents the collapse into near-identical latents seen in some runs with more latent tokens - it adds 8.2 points to Coconut on GPT-2. PMPS (Li et al. 2026) gets a comparable result without a decoder, by soft-aligning latent positions with the teacher's hidden states at explicit chain-of-thought token positions through a shared set of learned prototypes. No parsed reasoning steps or step labels are involved.

The teacher ceiling

A student trained to reproduce a written chain might be expected not to exceed it. The evidence is mixed. Coconut's 97.0% on ProsQA against 77.5% for the chain it was distilled from is a counterexample, and the same model loses to the chain on GSM8K. CoLaR's 8B model, after reinforcement learning, reaches 37.5% on GPQA against 35.7% for the chain-of-thought teacher (Tan et al. 2025). Distillation bounds the distribution of traces the student imitates rather than its accuracy, and SLPO (You et al. 2026) names the result: latent reasoners "remain largely imitation-bound, while explicit CoT has already moved past imitation via outcome-reward RL." A student inherits the teacher's errors along with its competence, and reinforcement learning is the route taken to get past them.

Aswal et al. (2026) train Coconut variants with the recurrence removed or the curriculum disrupted and find the breadth-first-like frontier structure, which had been read as evidence of latent search, in the controls as well. A curriculum that produces decodable structure in the hidden states has not thereby been shown to produce the computation that structure was taken to indicate.

The two routes combined

Few of the reinforcement learning methods below start from a base model. The usual arrangement is a supervised warm-up that teaches the model to produce latent steps at all, followed by a reinforcement stage that improves what those steps achieve. CoLaR trains its latent head by supervised compression of written chains and then runs GRPO on the result. LaDiR (Kang et al. 2025) fits its autoencoder and diffusion model to written reasoning steps, and LaDi-RL (Kang et al. 2026) takes that checkpoint as the starting point for reinforcement learning. SWITCH (Yang et al. 2026) trains with a visible-to-latent curriculum before its GRPO variant, and reports the curriculum-only model at 66.7% on MATH-500 against 79.3% after reinforcement. Dropout-GRPO (Jung 2026) and SLPO both begin from Coconut or CODI models that already went through the supervised recipes above. RLTT (Williams & Türeci 2026) starts from Ouro-Thinking, which is Ouro (Zhu et al. 2025) after a reasoning SFT stage.

The division of labor is consistent across these systems. The supervised stage supplies the latent representation and its position in the sequence, which outcome supervision alone does not determine, and the reinforcement stage supplies the exploration that distillation cannot, moving the model past the teacher's trace distribution. The methods that skip the supervised stage are the ones whose latent step is built from ordinary token distributions. Soft Tokens, Hard Truths (Butt et al. 2025), HRPO (Yue et al. 2025) and TARPO (Zhang et al. 2026) all start from an instruction-tuned model and run reinforcement learning directly, because a soft token or a gated hidden state needs no new representation to be learned first. HRPO makes this a stated motivation, arguing that trace-dependent training fails to use the reasoning the model already has.

Why a deterministic rollout has no policy

Outcome-reward reinforcement learning of the GRPO family needs three things from the model: several distinct rollouts per prompt, a log-probability for each, and a gradient of that log-probability with respect to the parameters. A discrete decoder supplies all three, because sampling a token from a softmax is a stochastic action with a tractable density. Coconut supplies none. The latent step maps a hidden state to a hidden state, so given the prompt and the weights every rollout is identical.

Jung (2026) states the consequence for group-relative methods: multiple rollouts of the same prompt yield identical latent trajectories, and under the low-temperature decoding the paper uses, identical answers too, so every reward in the group matches, the group-mean advantage is zero for each member, and training stalls. What the advantage needs is reward variance, which sampling the answer tokens at temperature could supply even from a fixed latent state. SLPO puts the same obstacle in terms of the likelihood. Intermediate steps "bypass the vocabulary distribution" so latent trajectories lack a tractable action likelihood, and a fixed thinking budget leaves nothing for an outcome reward to lengthen or shorten. There is no distribution $\pi$, no log-probability, and no score function, so a policy gradient has nothing to differentiate.

One could differentiate through the rollout instead. The latent chain is a composition of differentiable forward passes, so the answer's log-likelihood has a pathwise gradient all the way back to the first thought, and Coconut's own training already uses it. That route reintroduces the cost the reinforcement learning route was meant to avoid: the backward pass runs through the whole sequential chain, one pass per thought. The methods in the next section all make the other choice. They inject noise so that a score-function estimator exists, and they accept its higher variance in exchange for a gradient that touches only one step's computation.

Reinforcement learning restores teacher forcing

The cost of the score-function route is lower than it first appears. During the rollout the noisy latent at each step is sampled and recorded. At update time those recorded latents are data. The whole input sequence is known in advance, so the log-probability forward pass runs over every position in parallel, as an ordinary teacher-forced pass does, and the sequential dependence lives only in generation. The group-relative baseline absorbs much of the variance the score-function estimator adds. The supervised route costs $n+1$ sequential passes per example. The reinforcement route costs $G$ rollouts per prompt, and the rollouts parallelize.

Where the noise goes

Every method below constructs a stochastic action from a deterministic latent step. They differ in the space the noise is added to, which fixes the form of the log-probability and therefore which parameters the gradient reaches.

Gaussian noise on the embedding

Butt et al. (2025) begin from the input used by Soft Thinking (Zhang et al. 2025), the probability-weighted mixture of token embeddings at temperature $\tau$, and add isotropic Gaussian noise to it, with scale set at a third of the root-mean-square embedding norm. The noise defines a policy. The action at step $t$ is the noisy embedding, distributed as $\mathcal{N}(\bar{x}_t, \sigma^2 I)$ around the deterministic mixture, so its log-probability is $-\lVert \tilde{x}_t - \bar{x}_t \rVert^2 / 2\sigma^2$ up to a constant, and its gradient with respect to the parameters touches only the computation that produced $\bar{x}_t$. Training uses REINFORCE with a leave-one-out baseline and no reference chains at all, on Llama and Qwen models up to 8B.

On Llama 3.2 3B, soft-token training beats discrete-token training at pass@32 on all three mathematics benchmarks. At greedy pass@1 the authors call the two "broadly comparable", but the direction depends on the training set: trained on MATH or DeepScaleR, soft training is roughly 2 to 5 points behind, and trained on GSM8K it is ahead on MATH-500 and OlympiadBench. Out of domain, soft training moves the likelihood of the correct answer less than discrete training does, on average, without always leaving it where the base model had it. Qwen 2.5 3B trained on MATH goes from 4.67 to 5.49 in per-token negative log-likelihood on ARC, against 5.62 for discrete training. The best configuration in the paper trains with soft tokens and then infers with ordinary discrete tokens. That is a recipe for a better discrete reasoner, obtained by exploring in a continuous space during training, rather than a latent reasoner at deployment. SofT-GRPO's ablation adds a second qualification: on a 1.5B model, replacing its Gumbel noise with Gaussian noise on the embedding costs 10.7 points of pass@1, which its authors attribute to soft tokens landing outside the region of embedding space the pretrained model has seen.

Gumbel noise in the logits

Wu et al. (2025) diagnosed why untrained Soft Thinking underperforms. Fed a mixture, the model follows the highest-probability component almost exclusively, and the divergence between its next prediction and the prediction from that single token alone sits near zero. The remedy they test is randomness at the mixture itself, by Dirichlet resampling of the weights or by the Gumbel-softmax trick on the logits, and the Gumbel version beats ordinary token sampling on average across eight benchmarks at 32B scale. They leave the reinforcement learning to future work.

SofT-GRPO (Zheng et al. 2025) builds that algorithm. It reports that naive combinations of soft thinking with GRPO gain at pass@32 while losing badly at average accuracy, because of the difficulty of injecting stochasticity into a soft token and updating the policy accordingly. The method has two components. Gumbel noise goes into the logits rather than onto the embedding, so the perturbed distribution still lies on the simplex and the resulting input is a legitimate mixture of real token embeddings. And the policy density is the Gumbel density of the noise, recovered under the current parameters as the difference between the perturbed and unperturbed logits, which gives GRPO a ratio and attributes credit to the underlying output distribution. Across 1.5B to 7B models the method beats discrete-token GRPO by 0.13 points at pass@1 and by 2.19 at pass@32. The first of those is within noise. What that supports is better coverage at a 32-sample budget rather than a better single answer. Reading it as diversity is consistent with the Wu et al. finding that the model mostly follows one token, but the metric does not separate the two: no semantic trace statistic is reported. Jain & Rappazzo (2025) arrive at a similar method from the discrete side, extending GRPO to mixture-of-token generation with Dirichlet weights, and report gains on seven of ten Reasoning-Gym tasks for a 1.5B model together with higher hidden-state entropy through the chain.

Stochasticity from the token sampler

HRPO (Yue et al. 2025) observes that continuous latent methods conflict with the discrete nature of autoregressive generation and that trace-dependent training fails to exploit whatever reasoning the model already has. It therefore samples discrete tokens as usual, projects the previous hidden state through the language-model head to a probability vector, and mixes the resulting normalised embedding average into the sampled token's embedding through a learnable gate. The paper's reason for the detour is that raw hidden states sit off the token-embedding manifold. The gate is initialized so that the input is almost entirely the token embedding and allowed to admit more hidden state as training proceeds. The randomness comes from the sampler, the update is REINFORCE with group-relative advantages and a KL penalty, and the reward determines how much latent content the gate admits. A 3B model trained this way beats the strongest 7B retrieval baseline on the knowledge suite and matches the best 7B model on four of the STEM datasets.

Under the taxonomy on the families page this is a hybrid rather than a latent method. Hidden state crosses between steps, but the loop still closes through the vocabulary at every position, and the latent content is a side channel rather than the trajectory. TARPO (Zhang et al. 2026) makes the split a decision: a small action head reads the hidden state and samples, at each step, whether to emit a token or take a soft step, and the router's log-probability joins the token log-probabilities under a shared group-relative advantage. SWITCH (Yang et al. 2026) makes the arrangement explicit. Two ordinary tokens open and close a latent block, so the GRPO ratio is defined at every text position including the boundaries. The latent positions between them contribute no policy-gradient term at all. The abstract says the objective propagates gradients through the recurrence, but the implementation runs latent segments inside torch.no_grad() and the limitations section says the latent positions contribute only through the key-value cache they hand to the next text segment. So the latent representation is shaped by the curriculum, not by the reinforcement objective.

A generator that is stochastic by construction

CoLaR (Tan et al. 2025) adds a latent head that predicts the mean and standard deviation of the next compressed embedding and samples from that Gaussian, so the latent step has a density before any reinforcement learning begins. A supervised first stage trains the head on embeddings compressed by a randomly drawn factor, and a second stage runs GRPO with a correctness-only, group-normalized reward. The compression pressure is not in the reward, which is binary. It comes from how the advantage is applied: the group-relative value is averaged across the latent positions, so a shorter chain concentrates more credit per position and a longer one dilutes it. The paper's own ablation isolates this. Removing the averaging leaves the exploration in place and the accuracy gain roughly intact, but the chain length runs straight to the cap instead of contracting. With it, the reinforcement stage adds up to 5.4% accuracy on MATH while cutting the latent chain by 82.8%.

LaDiR (Kang et al. 2025) compresses each written reasoning step into a block of latent tokens with a variational autoencoder and generates the blocks by latent diffusion. Diffusion sampling is stochastic on its own account, and LaDi-RL (Kang et al. 2026) exploits that by running the denoising as a stochastic differential equation whose Gaussian transitions give GRPO its ratios. The paper's framing is entropy collapse: token-level GRPO on the same base improves pass@1 but never beats the base model at pass@128, while the latent-diffusion policy improves both. It also faces a credit-assignment problem of its own. The policy acts in latent space and the reward arrives after decoding to text, so a wrong answer could be a bad latent or a bad decode. The fix is to sample several latent trajectories and several text completions per latent, averaging the decodes for each latent's advantage. The two axes are not interchangeable. Four latents alone moves pass@1 from 60.0 to 68.5, four decodes alone only to 63.5, and the full four-by-four rollout to 77.1, so most of the gain belongs to the latent diversity rather than to averaging the decodes.

Noise in the weights

Dropout-GRPO (Jung 2026) draws one Bernoulli mask per rollout and holds it fixed across every latent step, so each rollout is a sample from a variational distribution over parameters and the group has something to be relative to. On a Coconut-style Qwen2.5-1.5B it moves GSM8K from 27.29% to 29.01%. SLPO uses the same source of randomness to fit a surrogate: several dropout passes give an empirical mean and variance for the next latent, and the realized latent is scored under that Gaussian rather than under the discrete dropout law that actually produced it. The paper is explicit that this is a surrogate, with propositions bounding when its gradient tracks the true one, and a correctness-supervised stopping head gives the outcome reward a horizon to lengthen. On Coconut with Llama-3.2-1B it adds 5.84 points at pass@8, and the paper reports longer latent computation being allocated to harder instances.

Credit assignment in looped models

Looped language models apply a shared block of layers $T$ times per token, so the latent reasoning is iterative refinement of a hidden state before each emission rather than a sequence of latent positions. They sample tokens normally, so the determinism problem above does not arise. A different one does. The authors of Ouro (Zhu et al. 2025), the largest looped models with open weights, report that their reinforcement learning attempts "did not yield significant performance gains over the final SFT checkpoint" and the released Thinking variants are supervised only. Their stated primary obstacle was engineering: the dynamic early-exit gate does not fit inference engines that assume a fixed execution path, and both the off-policy workaround and a fixed-depth variant fell short of the supervised checkpoint.

RLTT (Williams & Türeci 2026) offers an algorithmic diagnosis. Standard GRPO takes its log-probability from the terminal loop's distribution alone, treating each token as a single-step decision when the model actually computed a trajectory $h^{(1)} \to \cdots \to h^{(T)}$ to reach it, so all credit for the intermediate states has to arrive through the final readout. The fix uses a property of looped models. Every intermediate loop's hidden state can be pushed through the existing language model head to give a proper next-token distribution, so RLTT replaces the single terminal log-probability with a weighted sum of per-loop log-probabilities of the sampled token, weighted by exit probability, by a progressive schedule, or uniformly. The per-loop logits already exist, so the overhead is negligible, and nothing is decoded and nothing external grades it.

Against compute-matched GRPO on Ouro-1.4B and 2.6B, with identical rollout budgets, rewards and advantage normalization over ten seeds, the paper reports mean mathematics accuracy up by 5.8 points at 1.4B and 10.9 at 2.6B across MATH-500, AIME24, AIME26 and BeyondAIME. Trained on mathematics only, the 2.6B model's GPQA score goes from 19.7% to 38.4%, and the 1.4B model's from 14.7% to 26.8%. Responses get shorter with no brevity term in the reward, and entropy falls more steeply than under GRPO. The authors read the first as the model converging internally sooner once intermediate states are rewarded, and the second as confidence rather than collapse, on the evidence that pass@k does not fall. Unchanged pass@k rules out losing the chance of a correct answer within k samples; on its own it does not settle confidence against trajectory collapse.

Two important things to note. The statement that reinforcement learning had failed on looped models is RLTT's account of the state before it, resting on Ouro's own report, and Ouro attributes the failure primarily to inference infrastructure rather than to the objective. And RLTT's own description keeps the terminal reward backpropagating through the loops, so the per-loop terms are a better-conditioned objective, not a demonstration that a terminal signal cannot reach the recurrence. The bibliography entry states the second point.

Process rewards without a readable trace

A process reward model for explicit chain of thought reads the steps. A latent step has no text to read, so every process-level signal for latent reasoning needs a readout of some kind. Four are in use.

• Decode and grade externally. LSRL (Ren 2025) decodes each recurrent depth of a 3.5B Huginn model (Geiping et al. 2025) to text and scores the partial solution with GPT-4.1-nano as a judge, aligned to final-answer correctness, and discounts the per-depth scores into a single trajectory reward, so the credit is shaped rather than depth-indexed: GRPO still sees one advantage and one whole-sequence ratio. It adds 4.27 points on GSM8K from 500 training problems. The cost is a decode per depth plus two grader calls per trajectory.
• Read through the head the model already has. RLTT's per-loop log-probabilities are a process signal that needs no decoding, because the language model head applied to an intermediate state is a legitimate readout. This works only when intermediate states live in the same space as the final one, which is true of looped models and false of most continuous-thought methods.
• Train a reader. SIM-CoT's auxiliary decoder and the generative-reconstruction supervision of Chen et al. are the supervised-training analog: a decoder is trained to regenerate the written step from the latent, and the reconstruction loss is the reward. PMPS replaces the decoder with prototype assignment, so the alignment is many-to-many between latent positions and chain-of-thought token states of unequal length, and nothing is ever rendered as text.
• Marginalize the reader out. LaDi-RL scores a latent trajectory by the mean reward of several text decodings of it. That is not a step-level signal, but it rests on the same distinction between the latent and its decoding, which should not share one reward.

Each of these obtains a signal by installing a readout. The monitorability page covers the consequence: a readout trained to make latent states gradeable can also be optimized against.

What is unsettled

The best latent-trained models often infer discretely

Butt et al.'s strongest configuration decodes hard tokens at inference. SofT-GRPO's pass@1 advantage over discrete GRPO is 0.13 points. HRPO, TARPO and SWITCH keep the vocabulary in the loop by design. Across the noise-injection family, the demonstrated gains are in multi-sample coverage and, where it was measured separately, in token-distribution entropy, which are training-time properties, and the evidence that a continuous rollout at deployment beats a discrete one trained the same way is limited.

The noise-location results disagree across setups

Butt et al. report Gaussian embedding noise working on 3B and 8B instruction-tuned models. SofT-GRPO reports the same noise costing 10.7 points on a 1.5B distilled model against its Gumbel alternative. The base models, temperatures, noise scales, and training data all differ, so neither result rules out the other, and no compute-matched comparison across noise locations on one base exists.

The looped-model result is one paper on one family

RLTT's gains are on Ouro at two sizes with ten seeds, a larger seed count than most of this literature uses. It has not been replicated on Huginn or on a retrofitted looped model, and the length reduction and entropy decline it reports are the same signals that, in other settings, have been read as collapse rather than confidence.

Scale

Most of the reinforcement learning results here are on models from 124M to 3B parameters, and the supervised results are mostly GPT-2 and 1B. The exceptions are worth naming: CoLaR reports an 8B run, SofT-GRPO goes to 7B, and LSRL trains a 3.5B Huginn. The implementations page has the reasons cross-paper comparison is unreliable even within that range.