Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper investigates the biases in reward design estimators used in reinforcement learning from verifiable rewards (RLVR), revealing that traditional interpretations of reward signals conflate self-consistency elicitation with genuine reward design. Through a controlled tabular-GRPO simulator, the authors derive a precise decomposition of reward contributions, demonstrating that the reward-design fraction varies significantly with prior strength, ranging from 0.139 to 0.05. The findings challenge existing assumptions about reward design in RLVR, highlighting the dominance of elicitation in certain contexts and providing a framework for more accurate audits of published results.
Reward design in RLVR is often misinterpreted, with self-consistency elicitation dominating the signal in many scenarios, leading to significant biases in performance evaluations.
Reinforcement learning from verifiable rewards (RLVR) improves reasoning even when the reward signal is spurious -- assigning credit to the group-plurality answer rather than a ground-truth verifier. Practitioners commonly interpret naive = acc(TRUE) - acc(RANDOM) as the reward-design effect. We prove this estimand is systematically biased: it conflates self-consistency elicitation (sharpening the policy toward its modal answer via majority pseudo-reward) with genuine reward-design signal. Using a controlled tabular-GRPO simulator we derive an exact telescoping decomposition total = null + elicit + rd and measure each term across five prior-strength levels. The reward-design fraction of the naive estimator ranges from 0.139 at weak prior (ps=0.20) to 0.05 at strong prior (ps=0.80), with the elicitation term flipping sign at the self-consistency crossover. A pre-registered 2x2x2 factorial confirms non-additivity (interaction ratio 0.385; AxC effect -0.089). A points-vs-bounds pilot gate shows strong-prior regimes are point-identified while near-crossover regimes are only bounded. Re-audits of two named published results yield ELICITATION DOMINATED (elicitation share 0.98) and REWARD DESIGN DOMINATED (rd share 1.18) verdicts respectively, demonstrating the diagnostic value of the partition. We pre-commit to submit regardless of flip outcome; a non-flip is a finding of equal standing. We release a reusable one-command harness for any alignment paper to run the same audit.