Search papers, labs, and topics across Lattice.
This paper introduces Capability-Guided Compression (CGC), a framework that uses Sparse Autoencoder (SAE)-derived capability density maps to guide the allocation of compression budgets across transformer components, addressing the problem of capability-blind compression. Capability density, a novel scalar measure, combines feature breadth, activation entropy, and cross-input consistency of a component's SAE feature activation distribution. The authors theoretically prove and empirically validate that components with higher capability density exhibit lower structural redundancy and reach phase transition points at lower compression ratios, offering a pre-compression mechanism for component-level phase transition prediction.
Forget blindly pruning LLMs: this work shows you can use Sparse Autoencoders to identify and protect the most functionally important components during compression, leading to more robust models.
Large language model compression has made substantial progress through pruning, quantization, and low-rank decomposition, yet a fundamental limitation persists across all existing methods: compression budgets are allocated without any representation of what individual model components functionally encode. We term this the capability-blind compression problem and argue it is a root cause of two well-documented failures -- the insensitivity of perplexity-based evaluation to reasoning capability loss, and the abrupt phase transitions in model performance recently characterized by Ma et al. (2026). We propose Capability-Guided Compression (CGC), a framework that addresses this by using Sparse Autoencoder (SAE)-derived capability density maps to allocate differential compression budgets across transformer components. Capability density is a formally defined scalar measure combining the feature breadth, activation entropy, and cross-input consistency of a component's SAE feature activation distribution. We prove theoretically that components with higher capability density exhibit lower structural redundancy and reach their individual phase transition points at lower compression ratios, providing the first pre-compression mechanism for component-level phase transition prediction. Experiments on GPT-2 Medium confirm that capability density is statistically independent of Wanda importance scores (Spearman rho = -0.054, n = 384 heads), establishing it as a genuinely novel compression signal orthogonal to all existing importance metrics. We report a negative result on PPL-based compression comparison and provide a principled diagnosis identifying GPT-2 Medium as an insufficient test bed for the full CGC hypothesis. The theoretical framework, density formalism, and orthogonality finding constitute a foundation for capability-aware compression research.
