Search papers, labs, and topics across Lattice.
To overcome the limitations of rigid, pre-defined failure classifiers in complex environments, the authors introduce AutoTraceGT, a multi-agent framework that automates qualitative grounded theory to induce behavioral taxonomies directly from raw execution traces. This addresses a critical interpretability bottleneck for agent evaluation by establishing a principled, auditable trail from unsupervised trajectory data to theoretical failure modes. Across six corpora, the pipeline recovers 73–91% of known human-annotated failure modes, surfaces novel uncataloged behaviors, and forms a deductive feature space that outperforms zero-shot and few-shot LLM baselines on downstream failure prediction.
Borrowing a 60-year-old qualitative methodology from the social sciences allows automated agent pipelines to recover up to 91% of expert-identified failure modes while catching critical behavioral anomalies human evaluators missed.
Understanding agent behavior requires methods that scale to thousands of trajectories and surface new patterns in long, often unfamiliar tasks where pre-built classifiers fall short. We propose to bring grounded theory into agent trajectory analysis: a six-decade-old qualitative method from the social sciences, with a principled saturation criterion and an auditable trail from data to theory. We propose AutoTraceGT (Automated Trace analysis through Grounded Theory), the first multi-agent pipeline that automates grounded theory on agent trajectories. It iteratively performs open, axial, and theoretical coding until saturation, producing a behavioral taxonomy tailored to each task. Across six trajectory corpora, AutoTraceGT produces codebooks that recover 73-91 percent of the failure modes in human-annotated taxonomies and surface additional patterns that those taxonomies miss. The emergent theoretical narrative aligns with prior expert accounts. Used as a deductive feature space, the codebook outperforms zero-shot and few-shot LLM baselines on downstream failure prediction. These results suggest Grounded Theory offers a scalable analytic tool for ML researchers and agent developers studying what agents actually do.
