Search papers, labs, and topics across Lattice.
This paper investigates the impact of noisy data on Reinforcement Learning with Verifiable Rewards (RLVR) for training large language models. By rigorously re-verifying a previously claimed 100% noisy dataset and correcting contamination with clean data, the authors demonstrate that noise is indeed destructive to RLVR performance. Experiments on mathematical reasoning and Text2SQL tasks reveal that models trained on truly noisy data perform significantly worse (8-12%) than those trained on clean data, even with recent RLVR algorithm improvements.
Contrary to claims that RLVR can handle noisy data, this work reveals that current RLVR methods still suffer significantly from data quality issues, with performance dropping 8-12% when trained on truly noisy data.
Reinforcement learning with verifiable rewards (RLVR) has driven recent capability advances of large language models across various domains. Recent studies suggest that improved RLVR algorithms allow models to learn effectively from incorrect annotations, achieving performance comparable to learning from clean data. In this work, we show that these findings are invalid because the claimed 100% noisy training data is"contaminated"with clean data. After rectifying the dataset with a rigorous re-verification pipeline, we demonstrate that noise is destructive to RLVR. We show that existing RLVR algorithm improvements fail to mitigate the impact of noise, achieving similar performance to that of the basic GRPO. Furthermore, we find that the model trained on truly incorrect annotations performs 8-10% worse than the model trained on clean data across mathematical reasoning benchmarks. Finally, we show that these findings hold for real-world noise in Text2SQL tasks, where training on real-world, human annotation errors cause 5-12% lower accuracy than clean data. Our results show that current RLVR methods cannot yet compensate for poor data quality. High-quality data remains essential.
