Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces GracefulRL, a novel reinforcement learning framework designed to train language models to execute safe fallback policies when task-completion is unreliable, thereby mitigating costly errors. The core challenge of optimizing for fallback behavior without explicit annotations is addressed by reshaping the optimization landscape to promote graceful failure. The key result is that models trained with GracefulRL largely preserve task-solving capabilities while selectively employing fallbacks, demonstrating RL's ability to incentivize reasoning about capability limits.
Language models can be trained to proactively avoid costly errors by learning to "fail gracefully" directly from outcome-level rewards, without needing explicit safety annotations or verifiers.
Reinforcement learning makes language models better at solving tasks, but the same mechanism also rewards guessing. When success is the only way to earn reward, opting for a safe fallback is always worse than a long-shot attempt. When a wrong action can corrupt a database or ship broken code, that incentive can be costly. Therefore, we propose training models to fail gracefully: to preserve successful behavior whenever possible while learning to execute a safe fallback policy when acting would be unreliable. However, naively using policy gradients with intermediate reward for fallback behavior is brittle due to unfavorable optimization dynamics. We address this challenge with GracefulRL, which reshapes the optimization landscape to promote graceful failure. Importantly, this behavior emerges directly from outcome-level reward signals, without annotations specifying when to fall back or additional inference-time machinery such as a verifier. Across a range of tasks and scales, we train models that largely preserve their task-solving capability while selectively falling back when acting would be unreliable. More broadly, our results show that reinforcement learning can not only incentivize models to succeed, but also to reason about the limits of their capabilities and change course when they determine that success is out of reach.
CMU ML