Search papers, labs, and topics across Lattice.
This paper introduces two innovative strategies for enhancing reinforcement learning in Large Language Models (LLMs): Privileged Value Functions (PVF) and TETHER. These approaches address the limitations of traditional group-relative methods by providing token-level signals and adaptively balancing between group-relative and value-based baselines, respectively. Experimental results demonstrate that both strategies significantly improve performance on various reasoning tasks, outperforming standard value function baselines and competing effectively with GRPO methods.
Privileged Value Functions can inject crucial token-level signals into LLM reinforcement learning, leading to substantial performance gains over traditional methods.
Reinforcement learning algorithms for Large Language Models (LLMs) are largely distinguished by their variance reduction strategy. Group-relative methods like GRPO reduce gradient variance by sampling multiple rollouts per prompt, but provide only sequence-level credit. Training is also blocked by straggler rollouts, reducing throughput and increasing off-policyness. Learned value functions theoretically address both problems, providing token-level advantages without requiring large groups. However, additional infrastructure engineering challenges combined with the practical success of critic-free methods have made it difficult to justify their inclusion in RL pipelines. We propose two complementary strategies to improve the performance of value function RL: 1) Privileged Value Functions (PVF) which provide an elegant mechanism to inject additional task-relevant token-level signal without biasing the policy objective; 2) TETHER, a baseline that adaptively interpolates between group-relative and value baselines depending on the value function accuracy. Across several reasoning tasks, both strategies consistently improve over the standard value function baseline, and are competitive with or outperform mean-baseline GRPO.
Mila