Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The paper introduces ToolFailBench, a diagnostic benchmark designed to evaluate tool-use failures in language model agents across 1,000 diverse tasks in fields such as finance and medicine. By categorizing failures into Tool-Skip, Result-Ignore, Output-Fabrication, and Unnecessary-Tool-Use, the authors reveal significant discrepancies in tool utilization among various models, with the best model achieving only an 86.33% Clean Tool-Use Rate. This highlights that models with similar overall performance can exhibit vastly different behaviors in tool usage, underscoring the need for more nuanced evaluation metrics in assessing LLM capabilities.
ToolFailBench uncovers that even top-performing LLMs struggle with tool use, revealing a staggering 89-point accuracy gap between models on control tasks.
Tool calling is central to modern language model agents, but aggregate benchmark scores often hide where tool use fails. A model that never calls a needed tool and a model that calls the tool but ignores the result can look similar under final task accuracy. We introduce ToolFailBench, a diagnostic benchmark for measuring tool-use failures across 1,000 tasks in finance, medicine, law, cybersecurity, and real estate. Tool-required tasks return values the model wouldn't guess, forcing it to trust the tool while control tasks attach the same tools but should be answered directly. We label each trace with Tool-Skip, Result-Ignore, Output-Fabrication, and Unnecessary-Tool-Use, using a rule classifier and two LLM judges aggregated by majority vote. Across 19 headline models, the best reaches 86.33% Clean Tool-Use Rate, showing that faithful tool use is not saturated. More importantly, models with similar aggregate scores fail in different ways: most stay disciplined on no-tool controls, while Llama-3.1 models show an Always-Call pattern, and at the same parameter scale Llama-3.1-70B and Qwen2.5-72B differ by 89 percentage points on control-task accuracy. Tool-use evaluation should measure not only whether agents call tools, but whether they use tool outputs correctly and avoid tools when none is needed.