Search papers, labs, and topics across Lattice.
This study investigates the impact of experiential knowledge on the performance of large language models (LLMs) in tool use, identifying critical stages of knowledge acquisition, activation, and internalization. The authors find that instance-level knowledge significantly enhances tool-use performance, while abstract knowledge provides limited benefits; they also reveal that expanding the width of reasoning through parallel sampling is more effective than depth expansion for activating knowledge. The proposed Knowledge-Augmented Tool Execution (KATE) framework integrates these insights, leading to substantial improvements in performance across various model scales in practical applications.
Instance-level experiential knowledge boosts LLM tool use performance significantly more than abstract knowledge, reshaping how we approach knowledge integration in AI.
Large language models (LLMs) rely on tool use to act as autonomous agents, yet often fail in multi-step execution due to insufficient tool-related knowledge and ineffective knowledge activation. Therefore, we present a systematic study on how knowledge influences tool-use performance, covering the stages of knowledge acquisition, activation, and internalization. In the knowledge acquisition stage, we acquire and evaluate various forms of experiential knowledge, and our analysis shows that simple instance-level knowledge can already provide strong and reliable gains, while abstract intent-level knowledge offers limited benefits. At inference time, to activate knowledge, we find that prompting LLM to expand the depth of reasoning yields diminishing returns, whereas expanding the width of reasoning by parallel sampling with aggregation more effectively activates latent experiential knowledge. At training time, for knowledge internalization, post-training with knowledge-augmented data further improves performance, with reinforcement learning outperforming supervised fine-tuning. Based on these insights, we propose the Knowledge-Augmented Tool Execution (KATE), a knowledge-augmented tool execution framework that integrates experiential knowledge with reasoning-width-expanded inference and knowledge-aware training. Experiments on BFCL-V3 and AppWorld demonstrate consistent and substantial improvements over strong baselines across model scales. Our Code is available at https://github.com/hypasd-art/KATE.
