Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
ToolMisuseBench, a new offline benchmark, is introduced to evaluate tool-using agents' robustness to common operational failures like invalid arguments and interface drift. The benchmark features deterministic fault injection across diverse tool environments (CRUD, retrieval, file, scheduling) and explicit budgets for steps, calls, and retries. Experiments reveal that while schema-aware methods improve fault-specific recovery, overall success remains limited under strict authorization and hard failure conditions.
Even with strong language understanding, tool-using agents still struggle with operational failures, highlighting a critical gap in current agentic systems.
Tool using agents often fail for operational reasons even when language understanding is strong. Common causes include invalid arguments, interface drift, weak recovery, and inefficient retry behavior. We introduce ToolMisuseBench, an offline deterministic benchmark for evaluating tool misuse and recovery under explicit step, call, and retry budgets. The benchmark covers CRUD, retrieval, file, and scheduling environments with replayable fault injection. It reports success, invalid call behavior, policy violations, recovery quality, and budgeted efficiency. We release a public dataset with 6800 tasks and a reproducible evaluation pipeline. Baseline results show fault specific recovery gains for schema aware methods, while overall success remains limited under the released authorization and hard failure settings.