Search papers, labs, and topics across Lattice.
The paper introduces Hessian-Enhanced Token Attribution (HETA), a new attribution method tailored for decoder-only language models that combines semantic transition vectors, Hessian-based sensitivity scores, and KL divergence to address the limitations of existing methods. HETA aims to provide context-aware, causally faithful, and semantically grounded attributions by modeling token-to-token influence and second-order effects. Experiments on a new benchmark dataset demonstrate that HETA outperforms existing methods in attribution faithfulness and alignment with human annotations.
Existing attribution methods fail to capture the nuances of autoregressive LLMs, but HETA's novel combination of Hessian-based sensitivity and semantic transition vectors finally delivers faithful and human-aligned token attributions.
Attribution methods seek to explain language model predictions by quantifying the contribution of input tokens to generated outputs. However, most existing techniques are designed for encoder-based architectures and rely on linear approximations that fail to capture the causal and semantic complexities of autoregressive generation in decoder-only models. To address these limitations, we propose Hessian-Enhanced Token Attribution (HETA), a novel attribution framework tailored for decoder-only language models. HETA combines three complementary components: a semantic transition vector that captures token-to-token influence across layers, Hessian-based sensitivity scores that model second-order effects, and KL divergence to measure information loss when tokens are masked. This unified design produces context-aware, causally faithful, and semantically grounded attributions. Additionally, we introduce a curated benchmark dataset for systematically evaluating attribution quality in generative settings. Empirical evaluations across multiple models and datasets demonstrate that HETA consistently outperforms existing methods in attribution faithfulness and alignment with human annotations, establishing a new standard for interpretability in autoregressive language models.
