Search papers, labs, and topics across Lattice.
This paper introduces RREDCoT, a novel approach for segment-level reward redistribution in reasoning models that addresses the high variance associated with traditional Monte Carlo methods in Reinforcement Learning fine-tuning. By leveraging the model's own capabilities to approximate optimal reward redistribution, RREDCoT effectively emphasizes critical segments of Chain-of-Thought traces, improving the credit assignment process. Experimental results demonstrate that RREDCoT outperforms Monte Carlo sampling and various attribution methods, leading to more efficient training in long contexts.
Reward redistribution can significantly enhance reasoning model performance by focusing on critical segments of Chain-of-Thought traces without incurring extra computational costs.
Recent advancements in reasoning language models have been driven by Reinforcement Learning (RL) fine-tuning. Most often, these rely on the Group Relative Policy Optimization (GRPO) algorithm or modifications thereof to steer the models to produce Chain-of-Thought (CoT) traces. The final answer can only be verified, and the reward assigned, after the CoT trace is complete, making it a delayed reward problem. GRPO and its modifications correspond to Monte Carlo methods in standard RL, which are known to suffer from high variance. A possible solution to this problem is the redistribution of rewards through credit assignment, where segments of the CoT trace that are important for arriving at the desirable solution are emphasized by assigning a higher reward. While Monte Carlo sampling can be used to provide an unbiased estimate of intermediate state values, its computational overhead makes it unsuitable for train-time credit assignment in long contexts at high granularity. We introduce RREDCoT (Reward REDistribution for Chain of Thoughts), which utilizes the model itself to approximate the optimal reward redistribution without additional generation. We investigate the advantages of our method compared to MC sampling and several attribution methods. We further analyze several aspects relevant to the construction of the redistribution such as segmentation of CoT traces and state value estimation.
