Search papers, labs, and topics across Lattice.
This paper introduces LongRCA Bench, a benchmark designed to diagnose failures in long-horizon agent executions by providing a comprehensive dataset of 1,140 failed trajectories across five domains, each annotated with human labels for responsible roles and root-cause steps. The authors highlight that existing benchmarks inadequately address the complexity of long trajectories, revealing that the strongest baseline achieves only 13.2% exact root-step accuracy. By employing a novel training-free method called Root-Cause Trajectory Attribution (RCTA), the study achieves significant improvements, with 51.1% responsible-role accuracy and 24.1% exact root-step accuracy, underscoring the importance of distinguishing these two aspects in failure diagnosis.
LongRCA Bench reveals that diagnosing failures in long-horizon agents requires distinct metrics for responsible roles and root causes, with a new method achieving over 50% accuracy in role attribution.
When a long-horizon agent execution fails, outcome-level evaluation reveals the unsuccessful result but not where the decisive error entered the trajectory. Developers must then inspect the full execution to identify the responsible role and localize the earliest decisive root-cause step. Existing failure-attribution benchmarks largely focus on shorter traces, leaving diagnosis across hundreds of recorded steps underexplored. We introduce LongRCA Bench, comprising 1,140 failed trajectories across five domains without injected errors. It provides independently scored human labels for the responsible role and earliest decisive root-cause step. The median trajectory contains 145 steps, and the strongest baseline reaches only 13.2% exact root-step accuracy. We further present Root-Cause Trajectory Attribution (RCTA), a training-free method that retrieves candidate error steps from segment summaries and traces them to available earlier handoff instructions. Using the same backbone, benchmark instances, and scoring protocol, RCTA reaches 51.1% responsible-role accuracy and 24.1% exact root-step accuracy. These results highlight the need to evaluate responsible-role attribution and exact root-step localization as separate targets in long-trajectory failure diagnosis.
