Search papers, labs, and topics across Lattice.
This paper investigates the applicability of attribution-based explainability methods, commonly used for static classification tasks, to agentic AI systems where behavior emerges over multi-step trajectories. Through empirical comparison on static classification and agentic benchmarks (TAU-bench Airline and AssistantBench), the authors demonstrate that attribution methods, while stable in static settings, are unreliable for diagnosing execution-level failures in agentic trajectories. They find that trace-grounded rubric evaluation is more effective at localizing behavior breakdowns in agentic systems, revealing state tracking inconsistency as a key factor in failures.
Attribution-based XAI, effective for static models, falls apart when explaining agentic AI, highlighting the need for trajectory-level diagnostics to understand failures.
Over the last decade, explainable AI has primarily focused on interpreting individual model predictions, producing post-hoc explanations that relate inputs to outputs under a fixed decision structure. Recent advances in large language models (LLMs) have enabled agentic AI systems whose behaviour unfolds over multi-step trajectories. In these settings, success and failure are determined by sequences of decisions rather than a single output. While useful, it remains unclear how explanation approaches designed for static predictions translate to agentic settings where behaviour emerges over time. In this work, we bridge the gap between static and agentic explainability by comparing attribution-based explanations with trace-based diagnostics across both settings. To make this distinction explicit, we empirically compare attribution-based explanations used in static classification tasks with trace-based diagnostics used in agentic benchmarks (TAU-bench Airline and AssistantBench). Our results show that while attribution methods achieve stable feature rankings in static settings (Spearman $\rho = 0.86$), they cannot be applied reliably to diagnose execution-level failures in agentic trajectories. In contrast, trace-grounded rubric evaluation for agentic settings consistently localizes behaviour breakdowns and reveals that state tracking inconsistency is 2.7$\times$ more prevalent in failed runs and reduces success probability by 49\%. These findings motivate a shift towards trajectory-level explainability for agentic systems when evaluating and diagnosing autonomous AI behaviour. Resources: https://github.com/VectorInstitute/unified-xai-evaluation-framework https://vectorinstitute.github.io/unified-xai-evaluation-framework
