Search papers, labs, and topics across Lattice.
DenoiseRL is introduced as a reinforcement learning framework that trains reasoning models by recovering from their own failures, eliminating the need for strong teacher models or curated datasets. The method optimizes for recovery from incorrect reasoning traces, converting these errors into learning opportunities and improving exploration. Experiments show DenoiseRL outperforms on-policy RL baselines on mathematical and general reasoning tasks, demonstrating improved training efficiency and self-corrective behavior.
Forget expensive teacher models – DenoiseRL turns a reasoning model's own mistakes into a powerful, scalable training signal.
Reinforcement learning has become a central paradigm for advancing reasoning in large language models, yet most existing methods still depend on stronger teacher models or heavily curated difficult datasets, limiting scalable capability improvement. In this paper, we introduce DenoiseRL, a reinforcement learning framework that substitutes external supervision with recovery-oriented optimization over failures from weak models. Instead of relying on stronger supervision or carefully engineered data, DenoiseRL learns directly from incorrect reasoning traces by converting them into opportunities for improvement, making training more scalable and less dependent on external resources. This yields a richer and more diverse learning signal, improving exploration efficiency from imperfect model behavior. As a result, DenoiseRL improves reasoning performance and overall training efficiency while reducing the need for expensive data curation or stronger teacher models. Empirically, DenoiseRL consistently outperforms strong on-policy RL baselines across competitive mathematical and general reasoning benchmarks and promotes stronger self-corrective behavior as training difficulty increases, highlighting an effective and scalable alternative pathway for improving reasoning in large language models.
