Search papers, labs, and topics across Lattice.
This paper introduces ReSum, a novel framework that enhances long-horizon reasoning in Large Language Models (LLMs) by integrating self-summarization within Reinforcement Learning with Verifiable Rewards (RLVR). The method addresses the challenge of excessively long reasoning rollouts that can compromise coherence and exhaust context by allowing the model to manage its own reasoning trajectory. Experimental results demonstrate that ReSum not only stabilizes generation by reducing token-level entropy but also improves performance by an average of 4% while decreasing rollout length by 18.6%.
Self-summarization in LLMs can enhance reasoning coherence and reduce context exhaustion, leading to a 4% performance boost with shorter rollouts.
Reinforcement Learning with Verifiable Rewards (RLVR) is a central technique for improving long-horizon reasoning in Large Language Models (LLMs). However, existing RLVR methods often encourage unnecessarily long reasoning rollouts, which can degrade reasoning coherence and exhaust the available context budget. Existing approaches to long-context organization often depend on external mechanisms to organize rollouts, rather than enabling the model to manage its own reasoning trajectory. To address this limitation, we propose ReSum, a novel RLVR framework that enables LLMs to compress and organize their reasoning trajectories through self-summarization. Our pilot studies show that self-summarization stabilizes generation by lowering token-level entropy, and that introducing a ``summarization''phrase can substantially mitigate errors propagated from an incorrect rollout prefix. Motivated by these findings, ReSum adopts a summarization-aware adaptive rollout mechanism that contrastively evaluates whether self-summarization benefits the ongoing reasoning process. Specifically, when the model spontaneously triggers self-summarization, ReSum masks the summarization phrase to create a contrastive branch; for non-summarization positions, it instead randomly injects the phrase to create a matched branch. We further design a summarization-aware advantage to enable finer-grained comparison between contrastive rollout trajectories. Extensive experiments show that ReSum improves performance at an average of 4\% while reducing rollout length by 18.6\%.
