Search papers, labs, and topics across Lattice.
This paper introduces CAHP (Complementary Attention Head Pruning), a novel framework for pruning attention heads in Transformer models by framing head selection as a global graph-theoretical problem. By employing graph-based clustering and information-theoretic distance measures, CAHP identifies a diverse subset of complementary attention heads without the need for predefined sparsity levels, leading to more efficient model compression. Evaluations on SST-5 and MNLI benchmarks reveal that CAHP significantly outperforms existing methods, particularly in high-compression scenarios, while also mitigating the proximity bias seen in gradient-based pruning techniques.
CAHP reveals that pruning attention heads through a graph-theoretical approach can enhance model efficiency without sacrificing performance, outperforming traditional methods in high-compression settings.
The remarkable success of Transformer-based models in natural language processing stems from architectural scaling, which leads to a large number of parameters and hinders deployment in resource-constrained environments. While structured pruning offers a pathway to compression, existing state-of-the-art methods often rely on gradient-based importance ranking or stochastic gating, which suffer from instability, structural degeneration, and the need for extensive manual hyperparameter tuning. In this paper, we introduce CAHP (Complementary Attention Head Pruning), a novel post-hoc framework that redefines head selection as a global graph-theoretical problem. Rather than evaluating heads in isolation, CAHP utilizes graph-based clustering combined with information-theoretic distance measures to identify and preserve a topologically diverse subset of complementary attention heads. Without requiring a predefined sparsity level or pruning ratio, the framework automatically determines the number of selected attention heads across layers by identifying a diminishing marginal performance curve, where pruning additional heads leads to a sharp degradation in performance, as determined by the chosen polynomial degree. Extensive evaluations on the SST-5 and MNLI benchmarks, across different Transformer model scales, demonstrate that CAHP consistently outperforms competitive baselines, particularly in high-compression regimes. Furthermore, our structural analysis shows that CAHP avoids the "proximity bias" of gradient-based pruning methods, which tend to preserve heads mainly in layers close to the output, and instead retains a functionally critical set of attention heads in the model's intermediate layers.
