Search papers, labs, and topics across Lattice.
This paper introduces a scalable Kronecker-based approximation for efficient Hessian analysis in billion-parameter language models, circumventing the need to store the entire Fisher matrix. The method uncovers critical vulnerability patterns, revealing that value projection layers are particularly sensitive and exhibit strong cross-layer correlations, while other components show architecture-specific behaviors. Extensive experiments demonstrate that the approximation effectively predicts performance degradation and recovery, providing a valuable tool for optimizing large models through guided compression strategies.
Value projection layers are the Achilles' heel of billion-parameter models, revealing surprising sensitivity that could reshape compression strategies.
In this paper, we propose a scalable Kronecker-based approximation that captures cross-layer interactions without storing the entire Fisher matrix, enabling practical Hessian analysis for billion-parameter networks where full computation is infeasible. Our approach reveals consistent vulnerability patterns: value projection layers exhibit the highest sensitivity and strongest cross-layer correlations across multiple model families, while other components exhibit architecture-specific behaviors. Through extensive experiments on quantization, sparsification, inter-layer corruption, and post-corruption fine-tuning, we demonstrate that our approximation strongly correlates with both performance degradation and recovery. Our framework provides a practical, theoretically grounded tool for identifying fragile components in large models, opening new avenues for guided compression and optimization strategies, such as mixed-precision allocation, layer-wise sparsity, and adaptive low-rank decomposition across layers and even individual weight groups.
