Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
EvolveTool-Bench is introduced as a diagnostic benchmark to evaluate the software quality of LLM-generated tool libraries, moving beyond simple task completion metrics. The benchmark defines library-level metrics like reuse, redundancy, composition success, regression stability, and safety, along with a per-tool Tool Quality Score. Experiments comparing code-level and strategy-level tool evolution reveal significant differences in library health (up to 18%) between systems with similar task completion rates, highlighting the need for software-centric evaluation.
LLMs can achieve similar task completion rates using vastly different tool libraries, with some libraries exhibiting significantly better software quality than others, exposing hidden risks in current LLM agent evaluation.
Modern LLM agents increasingly create their own tools at runtime -- from Python functions to API clients -- yet existing benchmarks evaluate them almost exclusively by downstream task completion. This is analogous to judging a software engineer only by whether their code runs, ignoring redundancy, regression, and safety. We introduce EvolveTool-Bench, a diagnostic benchmark for LLM-generated tool libraries in software engineering workflows. Across three domains requiring actual tool execution (proprietary data formats, API orchestration, and numerical computation), we define library-level software quality metrics -- reuse, redundancy, composition success, regression stability, and safety -- alongside a per-tool Tool Quality Score measuring correctness, robustness, generality, and code quality. In the first head-to-head comparison of code-level and strategy-level tool evolution (ARISE vs. EvoSkill vs. one-shot baselines, 99 tasks, two models), we show that systems with similar task completion (63-68%) differ by up to 18% in library health, revealing software quality risks invisible to task-only evaluation. Our results highlight that evaluation and governance of LLM-generated tools require treating the evolving tool library as a first-class software artifact, not a black box.