Search papers, labs, and topics across Lattice.
To overcome the efficiency bottlenecks of decoupled pruning and quantization, this work formulates simultaneous model compression as a unified Bayesian variational learning problem termed Sparse Quantized Sub-distributions (SQS). The approach pairs a sparsity-inducing spike-and-slab prior with Gaussian Mixture Models to represent discrete low-bit weight clusters, solving the resulting intractable posterior through a theoretically consistent variational approximation. Evaluated across architectures spanning ResNet, BERT, Llama 3.2, and Qwen 2.5, SQS consistently achieves higher compression rates than sequential baselines while preserving baseline performance.
Jointly pruning and quantizing weights under a single Bayesian objective breaks the traditional trade-offs of sequential compression pipelines, squeezing modern LLMs like Llama 3.2 and Qwen 2.5 further without catastrophic accuracy degradation.
Compressing large-scale neural networks is essential for deploying models on resource-constrained devices. Most existing methods adopt weight pruning or low-bit quantization individually, often resulting in suboptimal compression rates to preserve acceptable performance drops. We introduce a unified framework for simultaneous pruning and low-bit quantization via Bayesian variational learning (\method), which achieves higher compression rates than prior baselines while maintaining comparable performance. The key idea is to employ a spike-and-slab prior to induce sparsity and model quantized weights using Gaussian Mixture Models (GMMs) to enable low-bit precision. Due to the intractability of the objective involving spike-and-slab priors with GMMs, we derive an efficient approximation that facilitates effective compression with minimal accuracy loss. In theory, we provide a consistent result for our proposed variational approach to a sparse and quantized deep neural network. Extensive experiments on compressing ResNet, BERT-base, Llama3.2, and Qwen2.5 models show that our method achieves higher compression rates than a line of existing methods with comparable performance drops. Project page: https://comeusr.github.io/SQS_Webpage.
