Search papers, labs, and topics across Lattice.
This paper reexamines the role of groups in critic-free reinforcement learning (RL) for large language models, identifying that their primary function is to mitigate false penalties on negative samples rather than just estimating value baselines. The authors introduce a novel approach called negative token filtering, which allows for stable training using a single rollout, thus addressing issues of data inefficiency and synchronization in traditional group-based methods. Their experiments demonstrate that this method achieves comparable performance on reasoning tasks and superior performance on agentic tasks compared to existing group-based RL techniques.
Negative token filtering enables stable single-rollout training, outperforming traditional group-based methods on agentic tasks while maintaining efficiency.
Reinforcement learning (RL) has become a central paradigm for post-training large language models. Existing critic-free RL methods typically generate a group of rollouts for the same question to estimate value baselines for advantage computation. However, this design suffers from data inefficiency, group synchronization barriers, and inflexibility with structured rollouts. In this work, we revisit the role of the ``group'' and show that its underlying function is not merely to estimate baselines but to prevent false penalties on negative samples. Building on this insight, we propose negative token filtering, a simple and effective strategy that enables stable single-rollout training. We apply it to two batch-level advantage methods, achieving comparable performance on reasoning tasks and stronger performance on agentic tasks relative to group-based RL techniques.
Mila