Search papers, labs, and topics across Lattice.
This paper identifies a critical flaw in Group Relative Policy Optimization (GRPO) where the advantage estimator can mistakenly reward rollouts that achieve correct answers through guessing rather than reasoning, a phenomenon termed "spurious advantage." The authors analyze three specific scenarios where this issue arises, leading to misleading policy behaviors that favor guess-like strategies. To address this, they introduce SIGNBALANCE, a new method that maintains a composition-free magnitude calculation, effectively improving performance on bounded-answer tasks and search agents while matching GRPO on open-answer benchmarks.
Guessing can masquerade as reasoning in GRPO, leading to misleading policy optimization that SIGNBALANCE effectively corrects.
Group Relative Policy Optimization (GRPO) is widely studied for reinforcement learning with verifiable rewards, where its advantage estimator assigns each rollout a magnitude from within-group reward statistics. In the common case, this magnitude rewards rollouts that reach the correct answer through reasoning. Yet, an overlooked case shares the same surface: a rollout may land on it by guessing, and the formula still assigns a high magnitude, which we identify as the spurious advantage. This arises in three cases: bounded-answer tasks with a small candidate set; open-answer sets hosting bounded sub-cases; and search agents whose budget opens many paths to the same answer. In all three, this misleads the policy toward guess-like behaviors. We propose SIGNBALANCE, whose magnitude is composition-free: it keeps the verifier sign, uses a global scale, and restores zero-mean balance via a stop-gradient per-class rescaling. Across math and search agent benchmarks at different scales, SIGNBALANCE matches GRPO on open-answer math and improves on bounded-answer math and search agents. Code will be released.
