Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces the Agent Error Dataset (AED), a collection of over 50,000 error-diagnosis pairs derived from text-based LLM agent rollouts across diverse environments and models, designed to facilitate failure analysis and error-aware post-training. The AED and an associated Agentic Error-to-Training (AET) pipeline enable the generation of diagnoses and concrete corrections for agent failures, which are then validated against recorded execution evidence. Experiments show that proposed corrections significantly improve verifier pass rates and that fine-tuning with diagnosed errors enhances agent performance on downstream tasks, outperforming success-only training.
LLM agents can improve their decision-making by learning from their own diagnosed failures, with first-proposal corrections boosting success rates by over 30 percentage points.
An unsuccessful LLM agent rollout contains more information than its final reward: the observations available to the agent, the actions it chose, and the environment's responses. Reusing this experience for learning requires identifying a decision to revise and testing a concrete alternative. We introduce the Agent Error Dataset (AED), comprising 50,228 error-diagnosis pairs from 9,961 source tasks across 33 environments, 19 harness families, and 23 policy models in text-based agent systems. We retain source traces and execution metadata to support cross-setting failure analysis and re-diagnosis without repeating the original rollout. Our five-stage Agentic Error-to-Training (AET) pipeline collects natural failures, generates diagnoses and proposed corrections, and checks them against recorded evidence. Where replay is supported, we compare corrections with original-action retries from the same checkpoint under matched execution settings. We then construct separate training views for diagnosis and actor recovery. Across 3,062 matched replay pairs, first-proposal corrections raise verifier pass rates from 18.4% to 51.1%, a gain of 32.7 percentage points. Using a separately frozen diagnosis release, full-diagnosis fine-tuning on 1,656 source tasks raises Qwen3-8B's exact-step agreement with internal teacher labels from 47.2% to 63.6%, averaged over three seeds on a 943-case holdout. The strongest prompted reference in this comparison scores 54.7%, and mean agreement improves at each of four increasing training-set sizes. In a single-seed comparison of actor-training recipes, action-only repair training scores 6.67 percentage points higher on WebShop-lite than success-only training.