Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
To eliminate the redundant prefill bottlenecks of context compaction in agentic RL, this work develops KV-streams, a method that continuously streams the key-value cache across compaction steps rather than flushing it. Beyond achieving a 2.6× to 5× wall-clock speedup across multiple compaction baselines without quality degradation, the persistent cache organically functions as a recurrent memory state. Crucially, the authors show that standard reinforcement learning alone is sufficient for agents to learn to store and retrieve information that has completely fallen out of the context window.
Context compaction in agentic RL can run up to 5x faster simply by streaming the KV cache instead of flushing it—accidentally turning standard LLMs into recurrent agents that preserve evicted context purely through RL.
Scaling the horizon of agentic LLMs is bottlenecked by the need to fit ever longer context traces in GPU memory. Context compaction has been the most popular mechanism to alleviate this issue, keeping GPU memory constant for a given trace. Unfortunately, most compaction strategies rely on prefilling the LLM context many times over, hindering training throughput. To alleviate this bottleneck and enable efficient trainable compaction, we propose KV-streams, a plug-and-play strategy compatible with any compaction strategy that substantially increases throughput while showing no evidence of hindering performance. KV-streams enable scalable compaction by streaming the KV cache forward rather than flushing it after each compaction. We show that KV-streams enable three different compaction strategies, achieving a 2.6 to 5x wall-clock speedup in training. Beyond efficiency, we find that the streamed KV cache can act as a recurrent state, carrying forward information that has long since disappeared from the context. Specifically, in a controlled setting we show that, contrary to prior work, RL alone is all that is needed for this behavior to emerge. Overall, we show KV-streams to be an efficient and lightweight plug-and-play addition to any post-training pipeline.
Mila