Search papers, labs, and topics across Lattice.
This paper introduces CVPO, a novel approach that enhances the reasoning capabilities of large language models (LLMs) through Curriculum-guided Value-Variance Policy Optimization. By addressing the issues of imprecise feedback and problem difficulty drift, CVPO employs a variance-aware advantage adjustment mechanism and a dynamic curriculum weighting method to optimize learning based on token-level value-variance and question difficulty. Experimental results demonstrate that CVPO significantly outperforms existing value-based baselines, leading to improved performance and exploration in complex reasoning tasks.
CVPO redefines LLM training by leveraging value-variance to enhance reasoning accuracy and exploration, outperforming traditional methods.
Reinforcement learning (RL) has emerged as an effective method for enhancing the reasoning capabilities of large language models (LLMs). However, existing methods suffer from insufficient precision in feedback on generated answer trajectories and exhibit the phenomenon of problem difficulty drift. To address these challenges, we propose CVPO - Curriculum-guided Value-Variance Policy Optimization. At the response trajectory level, we find that token-level value-variance correlates with exploration intensity. Our theoretical analysis shows this variance bounds policy update magnitude. We then use the estimated trajectory value-variance to quantify the intrinsic randomness in generation. Based on this, we design a variance-aware advantage adjustment mechanism for different reward types. At the question level, we introduce a dynamic curriculum weighting method that adapts to question difficulty. This helps the model focus on tasks matched to its current ability during each training stage. Experimental results show our method outperforms strong value-based baselines like VAPO. It achieves better performance and stronger exploration, enabling more accurate and robust reasoning in language models across various math tasks.
