Search papers, labs, and topics across Lattice.
This study investigates the instability of reinforcement learning (RL) in multi-step tool-use tasks for large language models (LLMs), revealing that catastrophic performance collapses are often triggered by unexpected probability spikes in control tokens. By systematically applying various supervisory signals, including off-policy supervision and hint-based guidance, the authors demonstrate that interleaving supervised fine-tuning (SFT) with RL can significantly enhance stability, although it may lead to performance degradation in out-of-distribution evaluations. The findings underscore the necessity of addressing RL failures and highlight how supervisory signals can facilitate robust exploratory learning in complex tasks.
Catastrophic collapses in tool-use performance can be mitigated by strategically interleaving supervised fine-tuning with reinforcement learning.
Tool use enables large language models (LLMs) to perform complex tasks, and recent agentic reinforcement learning (RL) methods show promise for enhancing model capabilities. However, RL alone often leads to instability or limited gains in tool-use tasks. In our experiments, some models exhibit catastrophic collapse, where performance abruptly drops and tool-invocation structures fail. The analysis reveals that these failures stem from unexpected probability spikes in specific control tokens, disrupting structured execution, yet the underlying tool-use capability remains intact, merely obscured by specific formats. To address this, we systematically investigate a diverse set of supervisory signals, including off-policy supervision, hint-based guidance, erroneous example supervision, and others, applied under both synchronous and interleaved training schemes. We find that interleaving supervised fine-tuning (SFT) with RL substantially improves stability, but exhibits degraded performance under format and content out-of-distribution (OOD) evaluation. We also analyze the impact of learning rates and generalization across settings. These results highlight the importance of understanding RL failures and demonstrate how diverse supervisory signals can guide exploratory learning, enabling robust training of LLMs for complex, multi-step tool-use tasks. Our Code is available at https://github.com/hypasd-art/Tool-RL-Box.
