Search papers, labs, and topics across Lattice.
The paper introduces Weakly-Supervised Group-Relative Policy Optimization (WS-GRPO) to improve the rollout efficiency of language models trained with GRPO for complex reasoning. WS-GRPO addresses the overthinking problem in GRPO by training a preference model from outcome-only correctness to generate prefix-level signals, guiding the model to continue or stop reasoning based on the perceived benefit of further continuation. Experiments on reasoning benchmarks demonstrate that WS-GRPO significantly reduces rollout length while maintaining competitive accuracy compared to GRPO baselines.
Stop wasting compute: WS-GRPO learns when an LLM should stop reasoning, slashing rollout lengths without sacrificing accuracy by turning final answer correctness into prefix-level guidance.
Group Relative Policy Optimization (GRPO) is effective for training language models on complex reasoning. However, since the objective is defined relative to a group of sampled trajectories, extended deliberation can create more chances to realize relative gains, leading to inefficient reasoning and overthinking, and complicating the trade-off between correctness and rollout efficiency. Controlling this behavior is difficult in practice, considering (i) Length penalties are hard to calibrate because longer rollouts may reflect harder problems that require longer reasoning, penalizing tokens risks truncating useful reasoning along with redundant continuation; and (ii) supervision that directly indicates when to continue or stop is typically unavailable beyond final answer correctness. We propose Weakly Supervised GRPO (WS-GRPO), which improves rollout efficiency by converting terminal rewards into correctness-aware guidance over partial trajectories. Unlike global length penalties that are hard to calibrate, WS-GRPO trains a preference model from outcome-only correctness to produce prefix-level signals that indicate when additional continuation is beneficial. Thus, WS-GRPO supplies outcome-derived continue/stop guidance, reducing redundant deliberation while maintaining accuracy. We provide theoretical results and empirically show on reasoning benchmarks that WS-GRPO substantially reduces rollout length while remaining competitive with GRPO baselines.
