Search papers, labs, and topics across Lattice.
This paper introduces ToolMaze, a benchmark designed to evaluate the dynamic replanning and anomaly recovery capabilities of LLM agents in the face of real-world tool failures, moving beyond traditional assessments that focus on idealized scenarios. The study reveals that performance significantly degrades under various perturbations, particularly with implicit semantic failures, leading to a 37% drop in Perturbation Recovery Rate (PRR) due to agents' over-reliance on corrupted outputs. Notably, while increasing model scale enhances fault-tolerance, it does so at a rate 3.66 times slower than basic task execution, indicating that dynamic replanning is a critical yet under-addressed challenge in LLM development.
ToolMaze reveals that LLMs suffer a staggering 37% drop in recovery performance when faced with implicit semantic failures, highlighting a critical vulnerability in current models.
Existing benchmarks evaluate Tool-Integrated Reasoning (TIR) in LLMs on idealized''happy paths'', largely overlooking real-world tool failures. We introduce ToolMaze, a benchmark for dynamic path discovery and error recovery in TIR agents. To separate systematic replanning from blind trial-and-error, ToolMaze adopts a two-dimensional design: DAG-based topological complexity and a $2 \times 2$ taxonomy of tool perturbations (explicit/implicit, transient/permanent). Evaluations show that perturbations degrade performance across nearly all models, with the sharpest drops under implicit semantic failures. Driven by systemic over-trust in corrupted outputs, Perturbation Recovery Rate (PRR) plummets by around 37\% in these scenarios, while complex topologies trap agents in futile trial-and-error loops. Crucially, agentic fault-tolerance improves with model scale $3.66\times$ slower than basic task execution, highlighting dynamic replanning as a distinct bottleneck unaddressed by model scaling or prompting. Data and code are available at https://github.com/Zhudongsheng75/ToolMaze.
