Search papers, labs, and topics across Lattice.
The paper introduces ToolMATH, a benchmark for evaluating tool-augmented language models on math problems requiring multi-step reasoning with diverse and overlapping tool sets. ToolMATH enables controlled evaluation of model reliability by manipulating tool availability and redundancy, revealing failure modes related to error accumulation and reasoning deficits. Experiments show that tool redundancy amplifies early errors, and while distractor tools can sometimes substitute for missing capabilities, they often lead to incorrect solutions.
ToolMATH reveals that long-range planning and coherent observation usage are more critical for tool-augmented LLMs than simply improving local action selection.
We introduce \ToolMATH, a math-grounded benchmark that evaluates tool-augmented language models in realistic multi-tool environments where the output depends on calling schema-specified tools and sustaining multi-step execution. It turns math problems into a controlled, correctness-checkable benchmark with tool sets, enabling systematic evaluation of model reliability under (1) large, overlapping tool catalogs and (2) the absence of the intended capability. \ToolMATH provides actionable diagnostic evidence of failure modes in tool-augmented agents, helping identify the control mechanisms required for robustness. \ToolMATH roughly contains 8k questions and 12k tools; we provide an additional hard-set \ToolMATHHard with questions and tools. Our evaluation reveals that the key failure factor is due to the inability to reason, leading to the accumulation of intermediate results' errors and constrain later decisions. Tool-list redundancy do not simply add noise, but amplify small early deviations into irreversible execution drift. The benchmark highlights that when the intended capability is missing, distractor tools can sometimes serve as partial substitutes in solution paths, yet they can also mislead models into ungrounded tool trajectories. Finally, comparisons between tool-use protocols emphasize that improvements come less from local action selection and more from long-range plan coherence and disciplined use of observations.
