Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces Multi-Session Agentic Reinforcement Learning (Mosaic) to address the mismatch between single-session training assumptions and multi-session inference in agentic RL, particularly for techniques like context compaction and subagent forking. Mosaic employs a critic-free hierarchical optimization approach that incorporates session-usage signals into advantage estimation at the trajectory level and shapes session advantages based on marginal contribution at the session level. Experiments on BrowseComp demonstrate Mosaic's superior performance over broadcasting methods in both context compaction and subagent forking, achieving better results with fewer sessions.
Agentic RL agents can now learn to efficiently manage multiple interaction sessions, outperforming naive broadcasting by precisely attributing value to individual session contributions.
Most existing agentic reinforcement learning methods assume single-session trajectories, where interaction proceeds within one continuously accumulated context. However, agentic techniques such as context compaction and subagent forking may produce trajectories spanning multiple sessions, creating a mismatch between the single-session training assumption and inference-time execution. Therefore, we introduce Multi-Session Agentic Reinforcement Learning, a more general setting that supports an arbitrary number of sessions within a trajectory. Under this setting, a widely adopted method is to broadcast trajectory-level advantages to all sessions, but this does not distinguish contributions from various sessions or guide session allocation within trajectories. To address this issue, we propose Mosaic, a multi-session agentic RL method with critic-free hierarchical optimization. At the trajectory level, it incorporates session-usage signals into advantage estimation. At the session level, it shapes each session's advantage according to the estimated marginal contribution. Experiments on BrowseComp with Qwen3.5-4B show that Mosaic outperforms broadcasting in both context compaction and subagent forking scenarios while using fewer sessions on average. Further analyses show that our method achieves improvements in session utility and allocation, which we suggested is a promising target for future advances in multi-session agentic RL.