Search papers, labs, and topics across Lattice.
This paper introduces Cliff, a novel reward shaping strategy for reinforcement learning with verifiable rewards (RLVR) that addresses the limitations of coarse outcome rewards by focusing on the first mistake in reasoning processes. By leveraging an off-the-shelf large language model (LLM) as a teacher, Cliff decomposes reasoning rollouts into correct and incorrect segments, providing token-level advantages that enhance learning efficiency. Experimental results across 12 scenarios show that Cliff improves reasoning performance by 15% over on-policy distillation and 7% over standard GRPO, even when using teachers with modest capabilities.
Cliff reveals that focusing on the first mistake in reasoning can significantly enhance the performance of reinforcement learning models, outperforming traditional methods.
Reinforcement learning with verifiable rewards (RLVR) has emerged as a powerful paradigm for large language model (LLM) post-training, but its reliance on coarse outcome rewards leads to limited guidance on intermediate reasoning processes. Existing approaches such as process reward modeling and on-policy distillation introduce additional constraints, such as reliance on a specialized reward model or assuming identical reasoning patterns between teacher and student. Nevertheless, we observe that once a reasoning process first goes wrong, evaluating the subsequent reasoning provides limited additional information, as it is already conditioned on an invalid prefix. Therefore, we propose Cliff, a reward shaping strategy that utilizes an off-the-shelf LLM as a teacher to identify the first mistake in each rollout. As a result, the rollout is naturally decomposed into two parts: a correct prefix and an incorrect suffix. Cliff then converts this signal into token-level advantages, assigning positive advantages for the correct prefix and negative feedback afterward. Experiments across 12 different scenarios demonstrate that Cliff consistently improves reasoning performance, outperforming on-policy distillation by 15% and standard GRPO by 7%, even with teachers of modest capability. Furthermore, we analyse the role of ``ground truth'' in Cliff and investigate its training dynamics. These results establish Cliff as a simple, general and effective approach for improving RLVR with richer, fine-grained supervision.
