Routing
Deciding what to compute next by reading the model's own state.
A router is a small function that reads something and picks an action: which expert, how many more loops, whether to stop, whether to emit a token or stay in latent space, which model answers at all. What it reads varies more than the literature admits, and the difference decides how much of this belongs on a page about latent reasoning.
The signal a router reads
Four tiers appear across the routing literature, and they are not equally latent.
| Signal | Where it is used | Note |
|---|---|---|
| Hidden state | Mixture-of-experts gates, early-exit classifiers, per-token depth routers, halting heads, latent-versus-explicit action heads | Dominant inside a single forward pass, where it is the only signal available. |
| Output logits | Cascades, speculative-decoding accept tests, confidence-gated escalation | Dominant across models, because it needs no access to internals. |
| Prompt text | RouteLLM, IRT-Router | Routes before generation begins. |
| Activations, across models | Prefill-activation routing, hidden-state speculative verification | Two confirmed papers, both 2026. Thin and unreplicated. |
Inside one model, routing on latent state is the only option, so the real comparison is routing against no routing, and routing wins on the compute-quality frontier. Across models, latent state is one option among several, and it is the least used.
Depth routing inside one model
Mixture-of-Depths (Raposo et al. 2024) introduced top-k token routing per layer: selected tokens get that layer's full computation, the rest take a cheap residual skip. Mixture-of-Recursions (Bae et al. 2025) applies the same idea to a shared recursive stack, so the router decides how many passes each token makes rather than which layers it visits. T-LoopFormer (Yu et al. 2026) routes loop-continue against loop-exit per token from the hidden state, with recursion-wise key-value caching.
Confident Adaptive Language Modeling (Schuster et al. 2022) is the antecedent for the exit-classifier form, reporting up to threefold speedups with calibrated sequence-level guarantees. A 2026 alternative dispenses with the learned gate entirely. One halts a token's recursion once successive loop outputs stop changing, measured by a divergence threshold, giving a 38% average depth reduction at quality parity on a 135M model without training a router at all.
Serving these models is a separate problem from designing them. Continuous depth batching (Schwethelm et al. 2026) reports 1.5 to 1.9 times higher offline throughput and 45 to 90% lower latency for Ouro and Huginn under dynamic load, closing most of the gap between the theoretical compute saving and what a batched server actually delivers.
Expert-choice routing
In the usual mixture-of-experts arrangement each token picks its top few experts, which leaves expert load uneven and needs an auxiliary balancing loss. Expert-choice routing inverts the direction: each expert takes its own top-$k$ tokens, so load is balanced by construction.
That balance belongs to a selector that is not causal. Ranking tokens jointly against a fixed capacity compares the current token's score against scores for tokens later in the sequence, which do not exist yet at autoregressive inference. Mixture-of-Recursions calls this information leakage and a “violation of causality during training.” Its Table 2 sets expert-choice's static compute budget against exactly that defect, where token-choice trades no leakage for load imbalance.
The top-$k$ oracle is unavailable at inference, so the deployed router approximates it: an auxiliary router or an auxiliary loss trained to predict whether a token would have been selected. Mixture-of-Recursions measures how well the approximation holds, reporting sampling accuracy alongside a dead-token ratio that exposes positional bias.
The inversion is what makes the mechanism portable to depth. If an expert is a recursion depth rather than a feedforward block, an expert-choice gate becomes a router deciding how many passes each token gets. What travels is the allocation idea. The guaranteed balance belongs to the noncausal training selector rather than to the deployed per-token decision.
It also has a consequence for what a transcript determines. When experts select tokens, a token's route depends on the rest of the batch, so the computation run on one sequence is not a function of that sequence alone. The same holds for token-choice routing under a hard capacity factor, where tokens reaching a full expert are dropped. The sufficient-statistic argument needs the batch held fixed as well as the prefix.
Mixture-of-experts as latent routing
A mixture-of-experts gate reads the token's hidden state through a learned linear map and softmaxes over experts. By the definition above, that is latent routing, and it is by far the most deployed instance of it.
Two 2026 results caution against reading much into what those routers decide. Wang, Hayou and Nalisnick bound the distance between two tokens' routing logits by their distance in the router-relevant subspace of hidden-state space, plus a residual. Similar hidden states therefore give similar logits. The converse fails, and the paper says so. Different hidden states can produce either similar or different logits, a relationship it calls many-to-one.
The bound is on logits rather than on the discrete route, which matters wherever a continuous representation is turned into a selection. A vector in the kernel of the gate leaves the logits untouched at any magnitude, and two states arbitrarily close together can still fall on opposite sides of a top-$k$ boundary and select different experts. What their empirical result shows is narrower: expert overlap across models on the same question runs near 60 per cent, comparable to the overlap between one model's answers to different questions. Routing tracks geometry, not domain expertise. Mohamud, Wagner and Ravanelli go further and show a fixed subspace of the raw hidden state routes competitively with no learned gate at all.
The two literatures do not read each other
Three dedicated latent-reasoning surveys from 2025 cite no mixture-of-experts routing work. The dedicated mixture-of-experts survey from the same year cites no latent reasoning, implicit chain of thought, or looped transformers. Coconut and the looped-transformer theory paper do not mention mixture-of-experts.
One paper bridges them. Mixture-of-Recursions repurposes expert-choice routing to select recursion depth, stating that each recursion depth becomes an expert, and cites both bodies of work by name. It is the field's only explicit statement that latent thinking depth and expert choice are the same mathematical object.
Two naming collisions will surface in any keyword search and belong in neither section. Chain-of-Experts is about sequential communication between experts inside a layer, not chain of thought. DeepSeek-V3's Multi-head Latent Attention uses latent for a compressed key-value cache, unrelated to latent reasoning.
Routing between latent and explicit steps
This is the part of the routing literature closest to the rest of this site, and it is very young. TARPO (Zhang et al. 2026) attaches an action head that reads the current hidden state at each generation step and samples a binary decision: emit a discrete token, or continue in continuous latent space. HyLaR (Cheng et al. 2026) does the multimodal version, interleaving text generation with continuous visual latents to avoid the semantic collapse that discretising an image causes, trained with separate clipping ranges for the textual and latent components. Its latent objective starts from a von Mises-Fisher policy, but the version it reports drops the normalisation, and the authors say the exact KL reading then no longer strictly holds.
Both appeared in 2026, within two months of each other. Neither reports numeric deltas in its abstract. The dial between token space and latent space that the rest of this site describes is, in these two papers, an actual learned decision rather than a design-time choice.
Routing between models
Cross-model routing is a large literature that mostly does not touch internal state. RouteLLM routes on query text and embeddings. FrugalGPT cascades on a learned scorer over question and answer text. Speculative decoding drafts from the target model's hidden features but makes the accept decision on output token probabilities.
Two 2026 papers are the exceptions. A prefill-activation router extracts residual stream activations before generation, reduces them by PCA, and predicts per-model correctness with an MLP, closing 45.58% of the gap to an oracle at 74.31% cost saving against the most expensive model, explicitly contrasted with text-only semantic routers. Dong et al. (2026) probe hidden states rather than logits to assess whether a drafted sequence is semantically plausible, reporting up to 2.7 times speedup.
That contrast is a real head-to-head. The text-only routers, some 1,300 configurations spanning embeddings, lexical features, perplexity and a fine-tuned classifier, run on the same model pools and the same held-out queries, and a paired bootstrap puts the AUC, Brier and cost-curve differences clear of zero in all three pools. What has not been run is a comparison against the output-confidence cascades under a shared compute budget, and nobody has replicated the result independently.
Label-conditioned classifiers as text routers
GLiClass and GLiNER2 score a query against a list of candidate labels in one encoder pass, and TypeSafe's Jev offers the same kind of decision through a hosted API. Any of them can pick a route, a tool or an escalation without generating text. They read the text they are given, not a running model's activations, so they belong in the prompt-text row of the table whatever their own activations do. Deciding without a written rationale does not show latent reasoning, and fixed-depth classification is still inference. The bibliography lists them with their antecedents, and the prediction page covers what their scores mean.
The GLiNER2 paper scores its classifier against GLiClass, a DeBERTa cross-encoder and GPT-4o on the same public sets. I found no comparison that puts Jev against the open encoders, or any of them against a reasoning model, on the same decisions. GLiClass times itself at batch size 1 on one GPU. TypeSafe reports agreement with a frontier-model reference on its own workflows, timed as calls to its service. A fair comparison would fix the input text, the label descriptions and the candidate set, including a none-of-these option. It would include a small generative model constrained to the same labels, since unconstrained generation is the easiest baseline to beat on cost. It would score against outcomes with a proper score, charge each system its full latency and cost at a stated batch size, and separate the gain from decomposing a task into several questions from the gain due to the model.
Probes as live gates
Most probing work reads internal state to analyse it. A smaller set uses the reading to decide something while generation is still running, and that is the case at issue here.
Zhang et al. (2025) probe reasoning-model hidden states to score intermediate answers, then simulate early exit by truncating completed traces at the first chunk the probe trusts. That cuts tokens by about 24% at 88.2% accuracy against 88.6% without exit, or 19% at no loss. The chunks come from parsing the finished trace, so the gate is never run during generation. Scalena et al. (2026) identify a commitment boundary, a step after which further chain-of-thought stops changing the greedily decoded answer. Their headline saving of 55% is a median over the traces where a boundary was found. Run online as a gate it saves 39% of tokens and drops MATH-500 accuracy from 80% to 71%, with a gentler setting saving 26% at 78%. Shu et al. (2026) run a single-layer probe token by token during decoding as a streaming moderation gate at sub-millisecond cost per token.
Adaptive halting for latent reasoning is the same mechanism pointed at loop count. Ouro computes an exit gate from the final-layer hidden state at each recurrent step under an entropy-regularised objective that stops it collapsing to maximum depth. Ning, Kuo and Gomes (2025) train a linear stop head by reinforcement learning and report a 52% drop in total reasoning length with no penalty to accuracy, though the abstract does not say which baseline that comparison uses.
A diagnostic study of Ouro's gates found that fixed-depth trajectories with a simple post-hoc confidence readout often match or beat the jointly trained gate, tracing the problem to how trajectories form under joint training rather than to gate capacity. The compute savings are real. The learned gate is not obviously the right way to get them.
AUROC
The probability that a randomly chosen positive example is scored above a randomly chosen negative one. A value of 0.5 is chance and 1.0 is perfect separation.
It says nothing about performance at any particular operating threshold, which is what a deployed gate actually needs. A probe at 0.99 AUROC can still be unusable if the false- positive rate at a tolerable recall is too high.
Failure modes
Probes are brittle off-distribution
Kumar (2026) reports deception probes at AUROC 0.998 or above on clean data, degrading sharply under stylistic distribution shift. The collapse is scale-dependent: the 27B model falls to 0.485, near chance, while 12B, 4B and 1B land at 0.609, 0.759 and 0.652. Probes trained with stylistic augmentation recover mean AUROC 0.979 at 4B and 0.983 at 27B on held-out styles. On clean data, a probe restricted to the first principal component reaches only 0.609 to 0.804, and five components give above 0.90. That does not show a multidimensional code. A logistic score on any number of components is still one direction in activation space, and the paper's permutation test finds no significant subspace ($k^* = 0$), which it reads as many weak features combined. The paper tests stylistic shift only, and says that whether stylistic and reinforcement-learning vulnerability are related failure modes remains open.
Probes decay when the model is updated
Duan (2026) finds activation monitors trained on a base model survive quantisation but frequently become unreliable after LoRA or QLoRA fine-tuning. A probe validated at deployment is not automatically valid after the routine model-lifecycle operations that happen continuously in production.
Much probe accuracy is surface reading
Boxó, Neelappa and Raval (2025) filter out tokens carrying explicit textual evidence of the target behaviour and probe performance drops by 10 to 30 AUROC points. Bias detection falls from 0.74 to 0.57. Sandbagging detection falls from 0.94 to 0.57 by the paper's abstract, though its results and conclusion sections both give 0.50, which they call chance. Much of what looks like reading a latent belief state is reading the same surface signal a text monitor would already have seen.
The paper limits that conclusion to linear probes and says so directly: sparse autoencoders, activation oracles and other white-box monitors were not tested and may not share the dependence. Reading it as a result about activation monitoring in general goes further than the evidence does. That bears directly on monitorability, where activation-level oversight is offered as the successor to reading the transcript. Part of its apparent independence is illusory.
Latent routers can be optimised against
Bailey et al. (2024) train a universal continuous embedding suffix, with white-box gradient access, that can take harmfulness probes from 100% recall to 0% at a 1% false-positive rate while keeping a 90% jailbreak rate. Gupta and Jenner (2025) show the policy itself can be trained to evade: reinforcement learning against a latent monitor produces evasion that partially transfers to unseen monitors, with token-level probes far more vulnerable than aggregated ones.