Search papers, labs, and topics across Lattice.
The paper introduces Context-Aware Layer-wise Integrated Gradients (CA-LIG), a novel framework for explaining Transformer models by computing layer-wise Integrated Gradients within each Transformer block and fusing these token-level attributions with class-specific attention gradients. CA-LIG addresses limitations of existing methods by providing context-aware, signed attribution maps that capture supportive and opposing evidence and trace the hierarchical flow of relevance through the Transformer layers. Experiments across diverse tasks and Transformer architectures demonstrate that CA-LIG provides more faithful, context-sensitive, and semantically coherent explanations compared to established explainability methods.
Finally, a way to understand *how* transformers make decisions, not just *which* tokens are important, by tracing relevance flow through layers with context-aware gradients.
Transformer models achieve state-of-the-art performance across domains and tasks, yet their deeply layered representations make their predictions difficult to interpret. Existing explainability methods rely on final-layer attributions, capture either local token-level attributions or global attention patterns without unification, and lack context-awareness of inter-token dependencies and structural components. They also fail to capture how relevance evolves across layers and how structural components shape decision-making. To address these limitations, we proposed the \textbf{Context-Aware Layer-wise Integrated Gradients (CA-LIG) Framework}, a unified hierarchical attribution framework that computes layer-wise Integrated Gradients within each Transformer block and fuses these token-level attributions with class-specific attention gradients. This integration yields signed, context-sensitive attribution maps that capture supportive and opposing evidence while tracing the hierarchical flow of relevance through the Transformer layers. We evaluate the CA-LIG Framework across diverse tasks, domains, and transformer model families, including sentiment analysis and long and multi-class document classification with BERT, hate speech detection in a low-resource language setting with XLM-R and AfroLM, and image classification with Masked Autoencoder vision Transformer model. Across all tasks and architectures, CA-LIG provides more faithful attributions, shows stronger sensitivity to contextual dependencies, and produces clearer, more semantically coherent visualizations than established explainability methods. These results indicate that CA-LIG provides a more comprehensive, context-aware, and reliable explanation of Transformer decision-making, advancing both the practical interpretability and conceptual understanding of deep neural models.
