Search papers, labs, and topics across Lattice.
This paper introduces Elastic Post-Training Sparsity (EPTS), a Multi-Sparsity framework that allows for the efficient compression of Large Language Models (LLMs) without the need for separate optimization sessions for each sparsity level. By utilizing a Multi-Sparsity Hierarchy LoRA (MS-HiLoRA) mechanism and a Multi-Sparsity Feature Mixer (MSFM), EPTS enables a single model to adaptively maintain performance across various sparsity configurations. Experimental results on LLaMA and OPT models show that EPTS not only competes with state-of-the-art methods like SparseGPT and Wanda but also significantly enhances deployment efficiency across diverse hardware scenarios.
A single elastic model can adapt to multiple sparsity configurations without re-optimization, revolutionizing LLM deployment efficiency.
Post-Training Sparsity (PTS) has emerged as a crucial paradigm for compressing Large Language Models to facilitate efficient deployment on resource-constrained devices. However, existing PTS methodologies are typically confined to Single-Sparsity optimization, necessitating a separate, time-consuming optimization session for each specific sparsity level. This rigid paradigm significantly hinders flexible deployment across diverse hardware scenarios, as adapting to a new sparsity requirement mandates a complete re-optimization process. To address these limitations, we propose Elastic Post-Training Sparsity (EPTS), a unified Multi-Sparsity framework that produces a single elastic model capable of maintaining robust performance across diverse sparsity configurations through a one-shot optimization process. Specifically, we design a Multi-Sparsity Hierarchy LoRA (MS-HiLoRA) mechanism that facilitates knowledge inheritance from low- to high-sparsity groups, effectively mitigating the competition for parameter reconstruction. Furthermore, we introduce a Multi-Sparsity Feature Mixer (MSFM), which significantly enhances the model's adaptability to pruning perturbations by dynamically fusing feature representations of varying sparsity granularities. Extensive experiments on LLaMA and OPT families demonstrate that EPTS achieves competitive performance compared to state-of-the-art methods like SparseGPT and Wanda, while offering significant efficiency gains by enabling multi-scenario deployment from a single optimization. our source code is available at https://github.com/xuke225/EPTS.
