Search papers, labs, and topics across Lattice.
The authors develop ToolEX, an automated framework that discovers functionally equivalent tool combinations to resolve the artificial one-to-one mapping bottleneck in tool retrieval benchmarks. Evaluating on the expanded ToolEQ benchmark reveals that 67.9% of sub-queries admit equivalent alternatives, expanding singular ground truths to an average of 5.3 valid tool combinations per query. Consequently, existing benchmarks systematically underestimate off-the-shelf retriever performance, revealing that 30–47% of reported fine-tuning gains are merely artifacts of incomplete annotation rather than true retrieval improvements.
Up to 47% of reported fine-tuning gains in tool retrieval are an evaluation illusion caused by benchmarks treating functionally equivalent tools as retrieval failures.
In open-world scenarios with massive and evolving tool repositories, tool-augmented large language models rely on a retriever to surface relevant tools for a given query. Because such repositories often contain many tools that implement the same functionality, a single query can often be resolved by several distinct but functionally equivalent tool combinations, making the natural query-to-tool mapping inherently one-to-many. However, existing tool retrieval benchmarks annotate each query with a single relevant tool combination, collapsing this one-to-many mapping into a rigid one-to-one annotation and causing valid retrieved tools to be misjudged as failures. To address this, we propose ToolEX (Tool Equivalent eXpansion), a framework that automatically discovers and annotates the tool combinations functionally equivalent to the labeled ones. Applied to the 7,360-query Tool-DE benchmark, ToolEX finds that 67.9% of sub-queries admit equivalent alternatives, expanding the singular ground truth to an average of 5.3 valid combinations per query. Using the expanded benchmark ToolEQ, we re-evaluate eight base retrievers and two fine-tuned variants; metrics on ToolEQ rise substantially over Tool-DE, showing that one-to-one annotation systematically underestimates retrievers and that 30--47% of the reported fine-tuning gain is an evaluation artifact rather than genuine improvement. Applying the same pipeline to skill retrieval on SkillRet further confirms that the one-to-one problem extends beyond tool retrieval.
Tsinghua AI