Search papers, labs, and topics across Lattice.
RaTA-Tool is introduced, a retrieval-based framework that enables Multimodal Large Language Models (MLLMs) to select appropriate tools for complex tasks by matching structured task descriptions derived from multimodal queries against machine-readable tool descriptions. This approach facilitates open-world tool use and extensibility to new tools without retraining. A preference-based optimization stage using Direct Preference Optimization (DPO) further enhances the alignment between task descriptions and tool selection, and a new dataset for open-world multimodal tool use is also introduced.
Forget training wheels: RaTA-Tool lets MLLMs pick the right tool for the job in the wild, even if they've never seen it before.
Tool learning with foundation models aims to endow AI systems with the ability to invoke external resources -- such as APIs, computational utilities, and specialized models -- to solve complex tasks beyond the reach of standalone language generation. While recent advances in Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs) have expanded their reasoning and perception capabilities, existing tool-use methods are predominantly limited to text-only inputs and closed-world settings. Consequently, they struggle to interpret multimodal user instructions and cannot generalize to tools unseen during training. In this work, we introduce RaTA-Tool, a novel framework for open-world multimodal tool selection. Rather than learning direct mappings from user queries to fixed tool identifiers, our approach enables an MLLM to convert a multimodal query into a structured task description and subsequently retrieve the most appropriate tool by matching this representation against semantically rich, machine-readable tool descriptions. This retrieval-based formulation naturally supports extensibility to new tools without retraining. To further improve alignment between task descriptions and tool selection, we incorporate a preference-based optimization stage using Direct Preference Optimization (DPO). To support research in this setting, we also introduce the first dataset for open-world multimodal tool use, featuring standardized tool descriptions derived from Hugging Face model cards. Extensive experiments demonstrate that our approach significantly improves tool-selection performance, particularly in open-world, multimodal scenarios.
