Search papers, labs, and topics across Lattice.
The paper introduces Gecko, a simulation environment that provides LLM agents with feedback on tool calls, including validity checks, response synthesis adhering to output schemas, and task objective assessment. This feedback enables iterative refinement of tool calls without the cost and risks associated with real-world tool execution. The authors demonstrate that using Gecko within a test-time scaling method called GATS consistently improves the tool calling performance of various LLMs, including GPT-4o, GPT-5, and Gemini-3.0-pro, on BFCLv3 and $τ^2$-bench benchmarks.
LLMs can significantly improve their tool-calling abilities by iteratively refining their calls within a new simulation environment, Gecko, that provides validity checks, realistic responses, and task completion assessments.
The ability to use tools is fundamental for large language model (LLM) agents. Given a task, existing systems use LLMs to plan and generate tool calls, which are executed by real-world tools to complete the task. However, tool calls are prone to errors because they are derived merely from LLM intrinsic capabilities. What is more, while it is useful to let LLMs iteratively refine the tool-call sequence using execution results from real tools, this process can be expensive and lead to unsafe results. To improve LLM tool calls and address issues caused by using real tools for refinement, we introduce Gecko, a comprehensive environment that simulates tool responses using a combination of rules and LLMs. Specifically, Gecko checks the validity of tool calls including input arguments and tool names, synthesizes reasonable responses that adhere to the output schema, and assesses whether all task objectives have been achieved. These three types of feedback provided by Gecko allow LLMs to refine their tool calls, forming a simple yet effective test-time scaling method named GATS. On BFCLv3 and $τ^2$-bench, GATS consistently improves the tool calling performance of various LLMs including GPT-4o, GPT-5, and Gemini-3.0-pro. We further discuss working mechanisms of our method and share future possibilities.
