Search papers, labs, and topics across Lattice.
This paper investigates continual reinforcement learning with verifiable rewards (RLVR) by introducing the Continual Reasoning Gym, a structured environment for evaluating text and visual reasoning tasks. The authors find that while sequential RLVR shows modest forgetting, it ultimately underperforms compared to multitask RLVR (MTRL), and they identify shared reasoning as a key factor that supports learning across tasks. By implementing Continual Prompt Replay (CPR), which reuses previous-task prompts to enhance performance on new tasks, the study demonstrates that CPR can achieve MTRL-level performance, highlighting its effectiveness in continual learning scenarios.
Shared reasoning structures can significantly boost performance in continual reinforcement learning, with a novel replay mechanism achieving parity with multitask training.
Reinforcement learning with verifiable rewards (RLVR) commonly post-trains reasoning models on multiple tasks, while rerunning multitask RLVR (MTRL) as new tasks are added makes capability expansion costly. We therefore study continual RLVR, which updates the existing model as each task arrives. The central question is whether a model updated this way can perform as well as a jointly trained model. To answer this question, we introduce Continual Reasoning Gym, a continual-RLVR environment that organizes text and visual reasoning tasks into five task sequences. In this setting, we identify two key observations: Sequential RLVR exhibits modest forgetting, yet its final performance remains below that of MTRL. To understand the latter, we decompose final performance and show that forgetting accounts for only part of the gap. To explain the former, we identify shared reasoning: transferable reasoning structure allows training on one task to support others on average. We therefore introduce Continual Prompt Replay (CPR), which harnesses shared reasoning to improve learning on the arriving and future tasks by replaying previous-task prompts and regenerating their responses with the current policy. On average, only CPR reaches MTRL-level performance.
