Search papers, labs, and topics across Lattice.
ToolOmni, a novel agentic framework, tackles the challenge of open-world tool use for LLMs by proactively retrieving relevant tools and grounding their execution within a reasoning loop. They fine-tune LLMs on a multi-turn interaction dataset and introduce a Decoupled Multi-Objective GRPO algorithm to optimize for both tool retrieval accuracy and execution efficacy. Experiments show ToolOmni achieves a 10.8% improvement in end-to-end execution success rate compared to existing methods, demonstrating enhanced robustness and generalization.
LLMs can now navigate the ever-expanding universe of external tools with significantly improved accuracy and generalization, thanks to a new agentic framework that proactively retrieves and grounds tool execution.
Large Language Models (LLMs) enhance their problem-solving capability by utilizing external tools. However, in open-world scenarios with massive and evolving tool repositories, existing methods relying on static embedding retrieval or parameter memorization of tools struggle to align user intent with tool semantics or generalize to unseen tools, respectively, leading to suboptimal accuracy of open-world tool retrieval and execution. To address these, we present ToolOmni, a unified agentic framework that enables LLMs for open-world tool use by proactive retrieval and grounded execution within a reasoning loop. First, we construct a cold-start multi-turn interaction dataset to instill foundational agentic capabilities via Supervised Fine-Tuning (SFT). Then, we introduce open-world tool learning based on a Decoupled Multi-Objective GRPO algorithm, which simultaneously optimizes LLMs for both tool retrieval accuracy and execution efficacy in online environments. Extensive experiments demonstrate that ToolOmni achieves state-of-the-art performance both in retrieval and execution, surpassing strong baselines by a significant margin of +10.8% in end-to-end execution success rate, while exhibiting exceptional robustness and generalization capabilities.
