Search papers, labs, and topics across Lattice.
This paper introduces TraceBench, a simulation-based framework designed for the controlled evaluation of LLM agents in time-series root-cause attribution tasks. By generating tasks from three interpretable mechanical systems, the study systematically assesses the performance of four LLM agents, revealing that these agents significantly benefit from domain context and primarily utilize numerical outputs over visualizations for analysis. Notably, the findings indicate that agents struggle more when tasked with producing Python scripts for root-cause labeling compared to direct predictions, highlighting critical insights into their operational capabilities in dynamic environments.
LLM agents excel in time-series analysis when given domain context, but their performance drops significantly when required to generate code for predictions.
LLM agents are increasingly applied to anomaly detection and root-cause analysis in time-series observations collected from real-world systems; however, their performance on these tasks has not been systematically evaluated under controlled conditions. We introduce TraceBench, a simulation-based framework for generating controlled root-cause attribution tasks. In each generated task, an agent receives time-series observations produced by simulating a physical dynamical system and must determine whether a system parameter was altered during the simulation and, if so, which one. Using TraceBench, we generate tasks from three interpretable mechanical systems and systematically evaluate four LLM agents across controlled experimental conditions, yielding new insights into how these agents analyze time-series observations from dynamical systems. Our results show that agents benefit substantially from domain context and explore data primarily through numerical console output rather than visualizations. We also find that agents generally perform worse when required to produce a Python script that maps each time-series sample to a predicted root-cause label than when they submit predictions directly. We release our datasets, agent trajectories, experimental results, and a leaderboard on our website, tracebench.github.io.
