Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces a post-tool execution reflection mechanism that leverages LLM-based reflection and domain-specific RAG to repair failed tool calls in agentic systems. The approach uses a combination of tool-specific documentation and troubleshooting documents to identify and correct both syntactic and semantic errors that are only apparent after the tool's response is analyzed. Experiments using the kubectl command-line tool for Kubernetes management demonstrate that the RAG-based reflection improves the execution pass rate by 55% and the correctness of answers to user queries by 36% on average, with troubleshooting documents outperforming official documentation.
LLMs calling tools get a 36% accuracy boost by reflecting on execution errors and retrieving relevant troubleshooting docs, not just official documentation.
Agentic systems interact with external systems by calling tools such as Python functions, REST API endpoints, or command line tools such as kubectl in Kubernetes. These tool calls often fail for various syntactic and semantic reasons. Some less obvious semantic errors can only be identified and resolved after analyzing the tool's response. To repair these errors, we develop a post-tool execution reflection component that combines large language model (LLM)-based reflection with domain-specific retrieval-augmented generation (RAG) using documents describing both the specific tool being called and troubleshooting documents related to the tool. For this paper, we focus on the use case of the kubectl command line tool to manage Kubernetes, a platform for orchestrating cluster applications. Through a larger empirical study and a smaller manual evaluation, we find that our RAG-based reflection will repair kubectl commands such that they are both more likely to successfully execute (pass rate) for 55% of our models evaluated and 36% more likely to correctly answer the user query on average. We find that troubleshooting documents improve pass rate compared to official documentation by an average of 10%.