← Home

Routing

Deciding what to compute next by reading the model's own state.

A router is a small function that reads something and picks an action: which expert, how many more loops, whether to stop, whether to emit a token or stay in latent space, which model answers at all. What it reads varies more than the literature admits, and the difference decides how much of this belongs on a page about latent reasoning.

The signal a router reads

Four tiers appear across the routing literature, and they are not equally latent.

The pattern is worth stating plainly. Inside one model, routing on latent state is the only option, so the real comparison is routing against no routing, and routing wins on the compute-quality frontier. Across models, latent state is one option among several, and it is the least used.

Depth routing inside one model

Mixture-of-Depths (Raposo et al. 2024) introduced top-k token routing per layer: selected tokens get that layer's full computation, the rest take a cheap residual skip. Mixture-of-Recursions (Bae et al. 2025) applies the same idea to a shared recursive stack, so the router decides how many passes each token makes rather than which layers it visits. T-LoopFormer (Yu et al. 2026) routes loop-continue against loop-exit per token from the hidden state, with recursion-wise key-value caching.

Confident Adaptive Language Modeling (Schuster et al. 2022) is the antecedent for the exit-classifier form, reporting up to threefold speedups with calibrated sequence-level guarantees. A 2026 alternative dispenses with the learned gate entirely: halt a token's recursion once successive loop outputs stop changing, measured by a divergence threshold, giving a 38% average depth reduction at quality parity on a 135M model without training a router at all.

Serving these models is a separate problem from designing them. Continuous depth batching (Schwethelm et al. 2026) reports 1.5 to 1.9 times higher offline throughput and 45 to 90% lower latency for Ouro and Huginn under dynamic load, closing most of the gap between the theoretical compute saving and what a batched server actually delivers.

Mixture-of-experts as latent routing

A mixture-of-experts gate reads the token's hidden state through a learned linear map and softmaxes over experts. By the definition at the top of this page, that is latent routing, and it is by far the most deployed instance of it.

Two 2026 results caution against reading much into what those routers decide. Wang, Hayou and Nalisnick bound the distance between two tokens' routing logits by their distance in the router-relevant subspace of hidden-state space, plus a residual. Similar hidden states therefore give similar logits. The converse fails, and the paper says so: different hidden states can produce either similar or different logits, a relationship it calls many-to-one.

The bound is on logits rather than on the discrete route, which matters wherever a continuous representation is turned into a selection. A vector in the kernel of the gate leaves the logits untouched at any magnitude, and two states arbitrarily close together can still fall on opposite sides of a top-$k$ boundary and select different experts. What their empirical result shows is narrower and still striking: expert overlap across models on the same question runs near 60 per cent, comparable to the overlap between one model's answers to different questions. Routing tracks geometry, not domain expertise. Mohamud, Wagner and Ravanelli go further and show a fixed subspace of the raw hidden state routes competitively with no learned gate at all.

The two literatures do not read each other

Three dedicated latent-reasoning surveys from 2025 cite no mixture-of-experts routing work. The dedicated mixture-of-experts survey from the same year cites no latent reasoning, implicit chain of thought, or looped transformers. Coconut and the looped-transformer theory paper do not mention mixture-of-experts.

One paper bridges them. Mixture-of-Recursions repurposes expert-choice routing to select recursion depth, stating that each recursion depth becomes an expert, and cites both bodies of work by name. It is the field's only explicit statement that latent thinking depth and expert choice are the same mathematical object.

Routing between latent and explicit steps

This is the part of the routing literature closest to the rest of this site, and it is very young. TARPO (Zhang et al. 2026) attaches an action head that reads the current hidden state at each generation step and samples a binary decision: emit a discrete token, or continue in continuous latent space. HyLaR (Cheng et al. 2026) does the multimodal version, interleaving text generation with continuous visual latents to avoid the semantic collapse that discretising an image causes, trained with separate trust-region constraints for the textual and latent components.

Both appeared in 2026, within two months of each other. Neither reports numeric deltas in its abstract. The dial between token space and latent space that the rest of this site describes is, in these two papers, an actual learned decision rather than a design-time choice.

Routing between models

Cross-model routing is a large literature that mostly does not touch internal state. RouteLLM routes on query text and embeddings. FrugalGPT cascades on a learned scorer over question and answer text. Speculative decoding drafts from the target model's hidden features but makes the accept decision on output token probabilities.

Two 2026 papers are the exceptions. An NVIDIA prefill-activation router extracts residual stream activations before generation, reduces them by PCA, and predicts per-model correctness with an MLP, closing 45.58% of the gap to an oracle at 74.31% cost saving against the most expensive model, explicitly contrasted with text-only semantic routers. A second probes hidden states rather than logits to assess whether a drafted sequence is semantically plausible, reporting up to 2.7 times speedup.

No head-to-head comparison exists between activation-based cross-model routing and the text-based or output-confidence alternatives on the same tasks. The claim that internal state helps rests on one recent unreplicated result.

Probes as live gates

Most probing work reads internal state to analyse it. A smaller set uses the reading to decide something while generation is still running, which is the case that matters here.

Zhang et al. (2025) probe reasoning-model hidden states to verify intermediate answers and exit early, cutting inference tokens by 24% without compromising performance. Scalena et al. (2026) identify a commitment boundary, a single step after which further chain-of-thought is causally inert, and early-exiting there shortens chains by up to 55% on average with negligible performance impact. Shu et al. (2026) run a single-layer probe token by token during decoding as a streaming moderation gate at sub-millisecond cost per token.

Adaptive halting for latent reasoning is the same mechanism pointed at loop count. Ouro computes an exit gate from the final-layer hidden state at each recurrent step under an entropy-regularised objective that stops it collapsing to maximum depth. Ning, Kuo and Gomes (2025) train a linear stop head by reinforcement learning and report a 52% drop in total reasoning length with no penalty to accuracy, though the abstract does not say which baseline that comparison uses.

A diagnostic study of Ouro's gates found that fixed-depth trajectories with a simple post-hoc confidence readout often match or beat the jointly trained gate, tracing the problem to how trajectories form under joint training rather than to gate capacity. The compute savings are real. The learned gate is not obviously the right way to get them.

Failure modes

Probes are brittle off-distribution

Deception probes reach AUROC at or above 0.998 on clean data and fall toward chance under stylistic distribution shift, adversarial suffixes, or reinforcement learning against the monitor. A single linear direction accounts for only 0.61 to 0.80 AUROC once the distribution moves, so the underlying structure is a subspace rather than a direction.

Probes decay when the model is updated

Activation monitors trained on a base model survive quantisation but frequently become unreliable after LoRA or QLoRA fine-tuning. A probe validated at deployment is not automatically valid after the routine model-lifecycle operations that happen continuously in production.

Much probe accuracy is surface reading

Boxó, Neelappa and Raval (2025) filter out tokens carrying explicit textual evidence of the target behaviour and probe performance drops by 10 to 30 AUROC points. Bias detection falls from 0.74 to 0.57. Sandbagging detection falls from 0.94 to 0.57 by the paper's abstract, though its results and conclusion sections both give 0.50, which they call chance. Much of what looks like reading a latent belief state is reading the same surface signal a text monitor would already have seen.

The paper limits that conclusion to linear probes and says so directly: sparse autoencoders, activation oracles and other white-box monitors were not tested and may not share the dependence. Reading it as a result about activation monitoring in general goes further than the evidence does. That bears directly on monitorability, where activation-level oversight is offered as the successor to reading the transcript. Part of its apparent independence is illusory.

Latent routers can be optimised against

Optimising activations directly drives latent-space defences from 100% recall to zero while retaining a 90% jailbreak success rate. Reinforcement learning against a latent monitor teaches evasion that partially transfers to unseen monitors, with token-level probes far more vulnerable than aggregated ones.