Search papers, labs, and topics across Lattice.
This paper introduces BENCH2ROBUST, a framework that transforms conventional tool-use benchmarks into stochastic environments that require agents to manage tool failures through retrying, switching, or abstaining. The study reveals a significant robustness gap in tool-using LLMs when faced with failures, demonstrating that Bayesian Tool Memory (BTM) can enhance robustness by up to 16.8 percentage points without retraining, while reinforcement learning (RL) strategies can learn effective recovery behaviors. Notably, the combination of BTM and RL achieves a remarkable improvement in robustness, reaching 40.8-45.5% under failure conditions while maintaining performance in failure-free scenarios.
Tool-using LLMs face a near-universal robustness gap, but combining Bayesian Tool Memory with reinforcement learning can boost recovery performance by over 40% in failure scenarios.
Tool-using LLM agents are commonly trained and evaluated in environments where tool calls succeed reliably, yet deployed tools can fail transiently, persistently, or silently. Robust recovery therefore requires more than repeated retries: an agent may need to retry the same path, switch to an alternative, or recognize that no viable path remains. We present BENCH2ROBUST, a framework that converts failure-free tool-use benchmarks into controlled stochastic environments with scenario-controlled solvability, where episodes explicitly require retrying, switching, or stopping after available paths are exhausted. We use BENCH2ROBUST to study two complementary interventions: structured runtime recovery context through Bayesian Tool Memory (BTM), and curriculum-controlled reinforcement learning. Across 7 models from 4 families and two multi-turn benchmark families, tool failures produce a near-universal robustness gap. On held-out Retail tasks, BTM improves robustness by up to 16.8 percentage points without retraining, while RL learns complementary recovery behavior that remains beneficial without inference-time BTM. Combining the two reaches 40.8-45.5% under injection while preserving failure-free performance. These results suggest that robust tool use benefits from combining environment-specific recovery knowledge with learned recovery behavior.
