Search papers, labs, and topics across Lattice.
This paper introduces Hi-TTRL, a novel framework for test-time reinforcement learning that addresses the sensitivity of reward signals to consensus strength in majority voting. By employing a Markov chain Monte Carlo (MCMC) hint sampler, Hi-TTRL regulates the consensus strength during rollouts, thereby mitigating the issues of low consensus amplifying unreliable updates and high consensus leading to vanishing gradients. Experimental results demonstrate that Hi-TTRL consistently outperforms standard TTRL across various datasets and architectures, validating the approach's effectiveness in enhancing reasoning capabilities of large language models.
Consensus strength in reinforcement learning can make or break model performance鈥擧i-TTRL offers a solution that fine-tunes this critical factor for better outcomes.
Test-time reinforcement learning (TTRL) improves the reasoning capabilities of large language models without labeled data by updating the policy with pseudo-labels constructed through majority voting. While effective, the reward signal assigned from majority voting is highly sensitive to consensus strength, defined as the frequency of the most common answer within a rollout group. In TTRL, consensus strength plays a dual role: it reflects both the reliability of the pseudo-label and the distribution of advantages. Low consensus can amplify updates from unreliable pseudo-labels through disproportionately large advantages, whereas high consensus reduces reward contrast and ultimately yields vanishing gradients. In this paper, we introduce Hi-TTRL, a test-time reinforcement learning framework that utilizes hints during sampling to regulate rollout consensus strength. Hi-TTRL first estimates consensus strength from a partial rollout group. When the consensus strength falls outside a target interval, it invokes a Markov chain Monte Carlo (MCMC) hint sampler. The sampler targets the power-transformed prefix distribution and uses finite-step approximate sampling to generate rollout prefixes as hints. By tuning the power exponent, Hi-TTRL generates hints with a sharpened or flattened power target, steering rollout consensus strength toward the target interval. Experiments on multiple datasets and backbones show that Hi-TTRL consistently improves over standard TTRL, with ablations and consensus-steering analyses validating the effectiveness of adaptive hint-guided consensus regulation.
