Search papers, labs, and topics across Lattice.
This paper introduces a contrastive attribution method, based on Layer-wise Relevance Propagation (LRP), to analyze LLM failures on realistic, long-context benchmarks by attributing the logit difference between incorrect and correct tokens to input tokens and internal states. They extend LRP to efficiently construct cross-layer attribution graphs for long sequences. Their systematic study reveals that token-level contrastive attribution provides informative signals in some failure cases, but its effectiveness varies across datasets, model sizes, and training checkpoints, indicating limitations in its universal applicability.
Token-level attribution struggles to pinpoint the causes of LLM failures in realistic settings, suggesting current interpretability tools may not be up to the task of debugging complex model behaviors.
Interpretability tools are increasingly used to analyze failures of Large Language Models (LLMs), yet prior work largely focuses on short prompts or toy settings, leaving their behavior on commonly used benchmarks underexplored. To address this gap, we study contrastive, LRP-based attribution as a practical tool for analyzing LLM failures in realistic settings. We formulate failure analysis as \textit{contrastive attribution}, attributing the logit difference between an incorrect output token and a correct alternative to input tokens and internal model states, and introduce an efficient extension that enables construction of cross-layer attribution graphs for long-context inputs. Using this framework, we conduct a systematic empirical study across benchmarks, comparing attribution patterns across datasets, model sizes, and training checkpoints. Our results show that this token-level contrastive attribution can yield informative signals in some failure cases, but is not universally applicable, highlighting both its utility and its limitations for realistic LLM failure analysis. Our code is available at: https://aka.ms/Debug-XAI.
Microsoft Research