Search papers, labs, and topics across Lattice.
This paper introduces LaMoC, a loss-aware modular compression technique that enhances parameter reduction in large language models (LLMs) by integrating activation statistics with Empirical Fisher statistics through gradient-error alignment. By addressing the gap in loss-sensitivity information, LaMoC reformulates joint modular compression as a two-tiered optimization problem, allowing for improved alignment between local module reconstruction error and downstream loss. The method demonstrates a 2.5% reduction in perplexity and a 1% relative improvement in task accuracy across various model families, outperforming existing modular compression approaches.
LaMoC achieves a surprising 2.5% reduction in perplexity while enhancing task accuracy, redefining the standards for modular compression in LLMs.
Modular compression has enabled considerable parameter reduction in LLMs while preserving strong language understanding and downstream task accuracy. However, existing joint modular compression methods primarily rely on activation statistics, leaving loss-sensitivity information and its module-level characterization underexplored. We investigate addressing this gap with LaMoC, a loss-aware modular compression methodology that blends activation and Empirical Fisher statistics through gradient-error alignment. LaMoC improves joint compression by selecting compression statistics that better align local module reconstruction error with the downstream loss. Our contributions are three-fold: (1) We characterize the Empirical Fisher as a module-level loss-aware proxy that can be blended with the activation statistics required for compression. (2) We reformulate joint modular compression as a two-tiered optimization problem that minimizes module reconstruction error while tuning the activation and gradient information blending rate. (3) We implement an empirically driven methodology with statistical validation to solve the resulting compression problem. We evaluate LaMoC across four model families spanning eight models. On the 4-8B models, LaMoC achieves an average 2.5% reduction in perplexity and a 1% relative improvement in task accuracy over state-of-the-art modular compression methods.
