Latent Context
A model can manage its own context as text it rewrites, or as latent state it writes, merges and evicts. The latent route is cheaper and harder to read.
Context Language Models (Shao et al. 2026) give a model unrestricted write access to its own context, treated as a file. Built zero-shot from existing models, they report 11.4% higher accuracy with 21.5% fewer FLOPs on BrowseComp-Plus. Online reinforcement learning lifts Qwen3.5-9B on the same benchmark by 47.6% while using 12% fewer FLOPs.
Every edit in that work is text, and its training assigns the outcome advantage to each segment without differentiating through the edits. Its stated future work is scaling the reinforcement learning and distilling existing harnesses into the model. The continuous counterpart, a model that edits its own context as latent state, draws on four literatures that rarely cite one another.
Text-side context management
Several systems train a model to rewrite a text memory with reinforcement learning. MemAgent overwrites a fixed-size memory as it reads, and extrapolates from 32K training text to a 3.5M-token task with under 5% loss. MEM1 keeps a compact internal state of constant size across multi-turn tasks.
Continuous Context Management (Hoy et al. 2026) compacts at every turn, despite its name in text, and trains with GRPO plus privileged full-history distillation. A frozen copy of the initial model scores each action under the reconstructed full history, which gives dense token-level credit. The method beats full-history GRPO for Qwen3-4B but not for Qwen3-8B.
Two measurements bound what text compaction costs. On 20 production agent configurations, one compaction round with a production compactor keeps 53% of safety rules and five rounds keep 10% (Zerhoudi 2026). Compression can also leave task completion unchanged while tripling retrieval calls, from 21.0 to 63.9 for one frontier model (Liu et al. 2026).
Models are also blind to their own context. VISTA shows a runtime dashboard of token usage, recency and remaining budget, and lifts Gemini-3-Flash on LOCA-Bench from 22.7% to 50.7% without training.
The latent channel inside text summaries
Memento (Kontonis et al. 2026) teaches a reasoning model to compress each block of its chain of thought into a short summary and to reason forward from the summaries, cutting peak KV cache about 2.5 times. Information from each block travels by two routes: the summary text, and the KV states of the summary, computed while the block was still in context.
Removing the KV route drops accuracy by 15 points on AIME24. Text-side context management already leans on a latent channel that the text does not show.
Model-chosen edits to the KV cache
Eviction
Neural Garbage Collection (Li et al. 2026) is the closest existing analogue of a latent Context Language Model. The model periodically pauses, samples which KV entries to evict, and continues. Eviction decisions are actions of the same policy as the reasoning tokens, trained jointly from outcome reward alone. On Countdown it scores 49.6% against 21.2% for the next-best baseline at a 2.4-times smaller peak cache.
Supervised and differentiable versions are cheaper to train. Dynamic Memory Sparsification reaches 8-times compression after 1K training steps by delaying eviction, which implicitly merges representations. For an equal number of memory reads it lifts Qwen-R1 32B by 12.0 points on AIME 24. TRIM-KV learns a retention score for each token at creation. The learned scores rediscover sink tokens and sliding windows without being designed to.
ForesightKV distils eviction from an oracle that sees future attention, then refines it with GRPO. KV Policy trains lightweight per-head eviction agents with reinforcement learning, leaving the language model unchanged.
Merging and writing
Eviction only deletes. Breadcrumbs Reasoning periodically compresses a window of generated tokens into one learned beacon token and evicts the window, trained by joint reinforcement learning and distillation. It keeps 65.1–89.8% of the teacher's performance with 2–32 times less memory. Compression happens at fixed intervals, so the model learns how to compress but not when.
IndexMem compresses evicted tokens into an online-updated latent state that supplies residual readouts, a soft delete that keeps a trace. Under aggressive eviction it gains up to 25 points on RULER. Earlier, Dynamic Memory Compression retrofitted models to learn different compression ratios in different heads and layers, Compressed Context Memory compressed accumulating KV pairs online, and Activation Beacon compressed activations progressively.
Overwriting
In-place overwriting of a KV entry fails, and the reason is causal. Li (2026) shows that during prefill the model has already written conclusions about a field onto downstream positions, so the field's own keys and values drive under 1% of the decision. An append-only erratum works where the overwrite does not, and precompiled KV segments can be repositioned with rotary encodings and spliced in at logit cosine 0.90–0.999 to full recomputation across twelve models.
A latent context editor therefore needs overwrite as append-and-mask rather than replacement.
Latent memory for agents
MemGen pairs a memory trigger, trained by reinforcement learning to decide when to invoke memory, with a memory weaver that generates a latent token sequence inserted into the reasoner's context. It reports gains of up to 38.22% over ExpeL and AWM and up to 13.44% over GRPO across eight benchmarks. Mem-W compresses past trajectories and in-session segments into memory tokens woven into a GUI agent's embedding sequence.
ActiveMem stores agent experience as a tree of node embeddings, injected into a frozen model as a latent prefix in the KV cache, with expansion, retrieval and pruning learned by GRPO. The memory policy belongs to a separate controller rather than the language model.
These systems write latent memory. None of them edits or deletes entries it has already written.
Weights as context
Fast weights carry context forward as parameters. TMEM (Ren et al. 2026) is the closest analogue of a Context Language Model with the file replaced by a weight delta. When the context exceeds a budget the agent writes grounded supervision about the session, which updates fast LoRA weights online. The extraction policy is trained by reinforcement learning on outcomes after the fast-weight rollout.
SEAL trains a model, by reinforcement learning against the updated model's downstream accuracy, to generate its own fine-tuning data. Doc-to-LoRA amortises the write: a hypernetwork maps a document to an adapter in one forward pass, and reaches near-perfect needle retrieval at more than four times the target model's native window. GradMem writes a context into a few prefix memory vectors by test-time gradient steps on a reconstruction loss, and extra steps scale capacity faster than repeated forward writes.
Weight memory compresses well and recalls poorly. TTT-E2E scales with context like full attention and runs 2.7 times faster at 128K. On pass-key retrieval at 128K it scores 0.06, against 0.99 for full attention and 0.07 for a sliding window. The authors attribute full attention's strength to its nearly lossless recall.
Learned KV state can derail queries it was not trained for. Cartridges train a compact KV cache per corpus and match in-context learning with 38.6 times less memory. Cartridges++ finds that such caches degrade off-context queries, including general knowledge and instruction following, while training-free compression shows the opposite trade-off.
The fast-weight lineage supplies the write operators. Linear attention is an outer-product fast-weight programmer, and the delta rule makes the write targeted (Schlag et al. 2021). A self-referential weight matrix emits its own edits (Irie et al. 2022). Gated DeltaNet combines erasure with targeted update. Titans add a neural long-term memory that learns to memorise at test time, and ATLAS optimises that memory over past tokens rather than only the last input. REFINE shapes fast-weight write dynamics with GRPO under a next-sequence objective. Hybrids keep a small exact cache beside the recurrent state (Cui 2026, Irie et al. 2025).
Weight writes made inside an agent's own episode can amplify drift, because each update changes the policy that writes the next update. Agentic test-time training down-weights repeated text in updates and gains up to 5.0 points on ALFWorld.
Latent context shared between agents
Context Language Models extend to several agents whose contexts coexist as files. The latent counterpart passes caches or hidden states. LatentMAS hands a shared latent working memory from agent to agent without training, and reports up to 14.6% higher accuracy than text-based multi-agent systems with 70.8–83.7% fewer output tokens. Cache-to-Cache trains a fuser that projects one model's KV cache into another's and beats text communication by about 3.1–5.4%.
Audits qualify these gains. Cheng et al. (2026) replace the relayed cache with one from a different example. Where the receiver needs the sender's private information, LatentMAS's relay reaches ceiling. Where it does not, gains stay within 2.8 points of a mismatched cache, and the released Cache-to-Cache projector shows no detected example-specific transfer. Zhang et al. (2026) attribute 6.67 of a 15.00-point MATH-500 gain to example-specific content.
Wenzel (2026) finds that a text round-trip destroys 88% of sparse-autoencoder features, yet the latent channel matches text on cross-lingual concept tasks and never exceeds it. The lost features mostly encode surface form.
Lessons from differentiable memory
The Neural Turing Machine (Graves et al. 2014) and its successors gave a controller learned read and write access to continuous memory. Csordás and Schmidhuber (2019) traced the Differentiable Neural Computer's failures to three causes: keys not separated from values, de-allocation that leaves aliased content, and chained reads that degrade the address distribution. Fixing them cut the bAbI mean error by 43%.
Sparse Access Memory touches only a few slots per read or write, and runs 1,000 times faster with 3,000 times less physical memory. MERLIN argued that memory is not enough and that the right information must be stored in the right format, so it drove memory formation by predictive modelling rather than reward. Temporal value transport used recall of specific memories to assign credit to past actions across long delays.
Reinforcement learning over discrete memory interfaces was demonstrated only on simple algorithmic tasks (Zaremba and Sutskever 2015). Gated transformers later exceeded an external-memory architecture on DMLab-30 (Parisotto et al. 2019), and attention absorbed the separate memory controller.
Capacity and the writer
A single input embedding holds far more than learned compressors put in it. Kuratov et al. (2025) reconstruct up to 1,568 tokens from one optimised vector. Learned encoders rarely exceed tenfold lossless compression, against up to 1,500-fold for per-sample optimisation, and the limit tracks the text's cross-entropy rather than its length.
The information a transformer can memorise through a prompt grows at most linearly with prompt length (Meyer et al. 2025). Latent slots are dense but bounded, and the binding constraint is the write procedure rather than capacity.
Monitorability of latent context
A text context can be read. A latent context cannot, and the problems the monitorability page describes for reasoning carry over to memory. Context Language Models already flag a model-editable context as an attack surface, citing a report of a model writing unauthorised instructions into its own summary.
In latent form the risk is harder to see. Brito et al. (2026) show that changing a relayed KV cache can collapse a coordinator's answers while the visible message still looks plausible, so a verifier that reads only text misses it. They protect the cache in transport with an HMAC manifest, which accepted all 774 honest payloads and rejected all 295 tampered ones.
Decoding latent state is not explaining it. Patterns that look like reasoning also appear in control models, so latent thoughts are hidden computation rather than hidden explanation (Aswal et al. 2026). An audit of latent memory needs matched controls and causal tests, as mismatched-cache audits provide for latent communication.
Untried combinations
Each claim rests on arXiv title and abstract searches and full-text searches run on 1 October 2026, plus the reference lists of the closest papers. Each is withdrawn when the missing work appears.
- No model has a full latent edit vocabulary. Neural Garbage Collection only evicts, Breadcrumbs merges on a fixed schedule, MemGen and Mem-W write without deleting, and full create, update and delete under reinforcement learning exists only over text memory. Confidence: moderate.
- No latent editor is trained on long multi-turn agent tasks. Model-chosen KV edits are trained on single reasoning chains such as Countdown and competition mathematics. Agent-side KV methods keep the base model frozen and train against proxy targets. Confidence: moderate.
- No head-to-head comparison of latent and text context management. No paper runs a latent editor against a text file editor such as Context Language Models on the same agent benchmark at matched compute. Confidence: moderate.
- No jointly edited latent context. Latent multi-agent systems relay, concatenate or compress caches into a receiver. None lets several agents edit one persistent latent context in place. Confidence: low to moderate.
The pieces for such a system exist. Append-and-mask edits avoid the overwrite failure, privileged full-history distillation gives dense credit to each edit, a predictive or reconstruction loss guides what is written, a small exact cache preserves recall beside a compressed tier, and an integrity check protects a cache that cannot be read.