Prediction

Latent reasoning where the answer is a probability or a number.

Most latent-reasoning work is evaluated on questions with a discrete correct answer: a multiple-choice letter, an integer, a path through a graph. A forecast is not like that. The output is a number in the unit interval, it is scored against an outcome that has not happened yet, and the score is continuous in the answer.

Why a probability is a different case

A model emits one token at a time from a vocabulary of order $10^5$, which is about seventeen bits per step. For a discrete answer that is a generous budget: the answer is one of a few options, and seventeen bits carries it. For a probability it is not obvious that the budget is the binding constraint either, since a forecast reported to two decimal places needs only about seven bits. The question is what happens to the intermediate quantities.

A calculation that ends in a probability passes through odds ratios, base rates, weights on conflicting evidence, and partial sums. Each is a real number. When reasoning is written out in tokens, every one of those intermediates is rounded to whatever the model chose to print, and the next step reads the rounded value rather than the one the model computed. Continuous latent reasoning removes that rounding step. Whether this matters in practice is an empirical question, and it has not been asked directly.

Training against a scoring rule

Reinforcement learning for reasoning almost always uses a binary correctness reward. Damani and colleagues note the side effect: such rewards do not penalise a confident wrong answer, so they degrade calibration. Their method, RLCR, has the model emit a numerical confidence after its reasoning and adds a Brier score to the correctness reward. They prove that this particular reward gives incentives for both calibration and correctness, and that for a general bounded strictly proper score there exists a large enough weight on correctness that the same holds. The paper is explicit that it does not extend to any proper scoring rule, and names log loss as a case that fails, and report that it “substantially improves calibration with no loss in accuracy” in and out of domain. Their summary of the contrast is direct: “while ordinary RL hurts calibration, RLCR improves it.”

Turtel and colleagues take the other route, training a 14B model with outcome-based reinforcement learning on questions drawn from a prediction market. The episode reward is the negative Brier score, which the paper names as “a strictly proper scoring rule that incentivises calibrated probabilistic forecasts”. The difference from RLCR is where the score is applied: here it scores the forecast itself against the resolution, rather than scoring a confidence attached to a separately judged answer.

Both operate on token-level reasoning with a verbalised number at the end. Neither trains a continuous latent trace.

Probes against the stated forecast

Sarfati and colleagues trained representation-pooling probes on the intermediate activations of a forecasting model and compared them with the model's own output. Three findings bear on latent reasoning directly.

The probes are better calibrated than the stated forecast, on Eternis-Forecaster 8B and on two GLM models. Removing an influential source from the prompt “often changes the model's forecast while leaving the reasoning trace untouched”, and the probe activations track that shift, predicting its direction in 84% of cases including where the written reasoning conceals it. Forcing an answer before any reasoning produces one that agrees with the model's eventual answer on 56 to 67% of questions depending on the split, with confidence correlating at 0.78 to 0.90. The authors read this as forecasts being largely fixed before reasoning begins, though agreement is not the same as precommitment. Routing on the spread of that early distribution saves 30 to 47% of generated tokens, with each split within about a point of full reasoning and confidence intervals including zero.

This is a measurement of the gap between what the internal state carries and what the text reports, taken on probabilities rather than on symbolic answers. It is one model family at 8B on one benchmark, and has not been replicated.

The sharpest version of the same comparison is not about forecasting at all. Yuchi, Du and Eisner probe hidden states for the magnitude of numbers a model has just read. A single linear projection recovers them to about 2.3% relative error on synthetic text and 19.06% on scientific papers, and a linear classifier ranks a pair of numerals at over 90% accuracy. Asked to rank the same pairs in words, the same models manage 50 to 70%. Their title puts it plainly: LLMs know more about numbers than they can say.

Reasoning and calibration

Whether producing a chain of thought improves calibration is contested, and the published results point both ways.

Yoon and colleagues compare six reasoning models with their non-reasoning counterparts across six datasets and find better calibration in 33 of 36 settings, attributing the gain to backtracking and exploring alternatives rather than to the presence of a chain as such. Mei and colleagues report the opposite for o3-Mini, DeepSeek R1 and Claude 3.7 Sonnet: verbalised confidence above 85% on wrong answers, with deeper reasoning making it worse, and introspection helping two of the three models and hurting the third. Hiremath and Hiremath describe a threshold beyond which additional reasoning budget induces overconfidence, though their own data do not show a clean trend, their validity rate is 42%, and the two calibration columns of their main table cannot both be right: expected calibration error is reported below the absolute overconfidence gap in every row, which the definitions do not allow. Zeng and colleagues find that reasoning-oriented training raises verbalised calibration while lowering abstention rates on factual questions, and that within a trace, shorter reasoning correlates with higher confidence, which is the opposite sign to Yoon.

The sign of the effect depends on the model family, on how much reasoning budget is spent, and on whether calibration is measured from the text or from a probe. Sarfati and colleagues' forced-answering result is relevant to that last point: a calibration number read off the chain of thought may be describing something the model had already decided.

Measured forecasting accuracy

ForecastBench scores models against expert forecasters and the general public on a rolling set of questions, and reports that expert forecasters outperform the best model at $p < 0.001$. PolyBench collects 38,666 prediction-market snapshots and scores seven models on the 5,097 that were resolved and had both an order book and news context. Models may skip a market, and decisions below 0.6 confidence are discarded, so the reported figure is a simulated confidence-weighted return on the trades each model chose to make. Two of the seven come out positive on that measure and five lose money.

Earlier work is consistent with this. Schoenegger and Park entered GPT-4 in a live Metaculus tournament against 843 participants and found it significantly less accurate than the human median, and statistically indistinguishable from answering 50% to everything. An ensemble of twelve models matched a 925-person crowd on 31 questions, 0.20 against 0.19 in mean Brier score. The preregistered test found no significant difference; the claim of equivalence rests on a separate test the authors did not preregister and whose bounds they call wide. Either way it is a result about aggregation rather than about any single model.

Numbers through a token vocabulary

Separately from reasoning, there is a body of work on how numeric quantities survive tokenisation, which bears on the same question one layer down. Singh and Strouse show that forcing right-to-left digit tokenisation substantially improves arithmetic in frontier models and that the error pattern looks algorithmic rather than random, though they are careful to say the mechanism is unclear. xVal replaces digit tokens with a single embedding scaled by the value, and reports better out-of-distribution generalisation on scientific data. Number Token Loss adds a proximity-aware term to cross-entropy on the grounds that cross-entropy treats digits as unordered categories, and matches a dedicated regression head while staying at token level. Song and Bahri compare decoding numbers as text against pointwise regression heads and find the two match, with the decoder additionally able to represent smooth densities.

Most of this concerns how a number is represented going in or coming out rather than how it is carried between reasoning steps. Not all: Singh and Strouse get much of their gain from prompting the model to rewrite the problem in the other tokenisation direction before answering, which is an intermediate step, and their control shows that answering in that direction without the rewrite does not work as well. xVal changes the input representation rather than the output layer.

Scores from classifiers that generate nothing

Label-conditioned encoders and decision APIs return numbers without writing them out, so nothing is rounded to printed digits, but what the numbers mean remains open. GLiClass and GLiNER2 apply a softmax across labels in single-label mode, giving a distribution over mutually exclusive alternatives, and an independent sigmoid per label in multi-label mode, giving scores that need not sum to one and specify no joint distribution. Neither activation is evidence of calibration. The GLiClass paper lists calibration differences across datasets among its limitations and reports no calibration metric.

TypeSafe's Jev returns, for a Choice question, a distribution over the options and a separate confidence, which the documentation defines as a statistic of the distribution's shape. In the Choice example the selected department has probability 0.61 and confidence 0.42, so confidence is not the probability that the selected answer is right. The same example describes the ticket as belonging to two teams, a multi-label fact carried by a single-choice distribution. The documentation leaves thresholds to be tuned on the user's own data.

The introduction says every question is evaluated “in parallel and in isolation against the same state”. That is a property of the execution, not of the events. If $P(A \mid x) = P(B \mid x) = 0.6$ exactly, then $P(A \cap B \mid x)$ can be anything from 0.2 to 0.6, and the product 0.36 requires conditional independence. Code that multiplies answers to separate questions assumes it. Calibration, $E[Y \mid p] = p$, is a property of a population, and a constant base-rate forecast has it while separating nothing. Guo et al. (2017) fit their recalibration on a hold-out validation set and assume it is drawn from the same distribution as the test data. A deployment shift breaks that assumption, and a training objective named for calibration does not restore it.

Latent reasoning on structured data

Dudley and Oymak apply latent chain of thought to time-series and tabular prediction. A structured-data transformer compresses its query-position hidden states into feedback tokens, appends them to the input, and reprocesses, giving several rounds of latent computation before it predicts. Across 36 datasets this improves on the baseline for 7 of 9 time-series datasets, averaging 12.63%, and 23 of 27 tabular datasets, averaging 3.25%.

The targets are distributional. Forecasts are decoded into quantiles and trained with the quantile loss at levels 0.1, 0.3, 0.5, 0.7 and 0.9, and tabular classification is trained with cross-entropy, which is itself a proper score. The comparisons are against a same-depth model without the chain, a deeper model matched on effective depth, and a weight-tied looped transformer. What is missing is a comparison against the same reasoning written out in tokens, so the result does not separate the benefit of extra computation from the benefit of keeping that computation continuous, and the paper reports no calibration diagnostics for the forecasts it produces.

What has not been done

Less than I thought when I started reading. Two of the obvious gaps turn out to be filled. Dudley and Oymak already run latent chain of thought against distributional targets, trained with quantile loss and cross-entropy, both proper scores. Yuchi, Du and Eisner already show an internal representation more precise than the printed one. Neither result is framed as being about latent reasoning and calibration, which is why they are easy to miss, but they are the results.

What remains is narrower. Nobody has compared a latent chain of thought against the same reasoning emitted as tokens, on the same distributional target, which is the comparison that would isolate the benefit of keeping the intermediate quantities continuous from the benefit of simply spending more computation. Dudley and Oymak have the target but not that baseline.

Nobody has measured how much numeric precision a decoding step costs. The argument that cross-entropy treats digits as unordered categories is made in the Number Token Loss paper, without a figure in bits attached to it.

And nobody has trained the intermediate steps to stay continuous with a proper score on the final distribution as the objective. RLCR scores a verbalised confidence after token-level reasoning. Turtel and colleagues score the forecast itself, also after token-level reasoning. Dudley and Oymak keep the reasoning continuous but do not report calibration. The three pieces exist separately.