Search papers, labs, and topics across Lattice.
This paper investigates the phenomenon of overthinking in RL-trained reasoning models, where unnecessary reasoning continues even after a correct answer is generated. By framing overthinking as a credit-assignment issue during GRPO-style reinforcement learning, the authors identify that successful trajectories often exhibit higher overthinking than unsuccessful ones, leading to a feedback loop that exacerbates the problem. The proposed Dynamic Rollout Editing (DRE) method effectively mitigates overthinking by preserving the correct reasoning prefix while editing out unnecessary continuations, resulting in improved model performance across various tasks.
Overthinking in RL models can amplify during training, but Dynamic Rollout Editing effectively curtails this behavior, enhancing reasoning efficiency.
Long-form chain-of-thought reasoning can improve LLM performance on complex tasks, but models often continue generating unnecessary reasoning after a correct answer has emerged. We refer to this behavior as overthinking. We study this phenomenon from the perspective of GRPO-style reinforcement learning (RL) post-training, framing it as a training-time credit-assignment problem rather than merely a decoding-time stopping problem. In rollouts sampled at the onset of GRPO training, we observe that successful trajectories can exhibit a slightly higher degree of overthinking than unsuccessful trajectories for the same prompts. This early imbalance provides a starting point for an undesirable feedback loop: because GRPO assigns sequence-level credit, it cannot distinguish the solution-reaching prefix from the unnecessary continuation that lengthens a successful trajectory. Both receive positive update signal, allowing the initial imbalance to grow into more severe overthinking during training. To address this issue, we introduce Dynamic Rollout Editing (DRE), a training-time intervention for successful trajectories that continue thinking after answer emergence. DRE preserves the accepted verified prefix, edits the remaining thinking, and prefers the edited trajectory within the same RL group, weakening the preference signal for unnecessary thinking without penalizing the reasoning needed to reach the answer. Experiments across diverse tasks show the effectiveness of DRE.
