Search papers, labs, and topics across Lattice.
This paper introduces CoRE (Consensus Rewards via Equilibrium), a novel approach for deriving rewards in test-time reinforcement learning by leveraging a graph-based representation of model roll-outs that incorporates answer agreement, reasoning similarity, and generation confidence. Unlike traditional majority voting, which can discard correct minority answers, CoRE refines the reward signal by extracting a dominant set through replicator dynamics, leading to a more nuanced and calibrated reward structure. The results demonstrate that CoRE significantly outperforms majority voting methods, achieving an average improvement of +21.7 points across various benchmarks and reducing the number of steps to reach plateau accuracy by 54-70%.
Transforming test-time reinforcement learning from a simplistic voting mechanism into a sophisticated consensus-based reward system could redefine how we approach model evaluation and training.
On unlabeled test data, reinforcement learning lacks a ground-truth reward; test-time RL methods derive one from the model's own roll-outs, rewarding those that match the majority vote over $N$ sampled answers. That vote discards a correct answer whenever it is a minority and scores every majority-matching roll-out identically. We replace it with \emph{CoRE} (Consensus Rewards via Equilibrium): the $N$ roll-outs form a graph whose edges combine answer agreement, reasoning similarity, and generation confidence, and replicator dynamics extract its dominant set, yielding a refined pseudo-label, a graded per-roll-out reward, and a per-question cohesiveness gate. CoRE strictly generalizes voting: majority voting is recovered as a special case; a block-value analysis gives a sharp threshold for when consensus recovers a correct minority against a larger wrong plurality; and confidence calibration provably lowers that threshold multiplicatively. Across seven backbones and five benchmarks (42 model--benchmark cells, three seeds each), \emph{CoRE} improves the untrained base by $+21.7$ points on average versus $+20.4$ for majority-vote TTRL, wins wherever agreement is contestable with margins over the vote of up to $+7.5$ points, and reaches the voting baseline's plateau accuracy in $54$--$70$\% fewer steps. Consensus, not counting: treating the roll-out group as a graph rather than a ballot box turns a brittle vote into a calibrated, graded, self-supervised reward at no extra roll-out cost.
