Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The paper introduces WiSparse, a training-free activation sparsity method for efficient LLM inference that addresses the limitations of existing approaches by incorporating weight information and accounting for inter-block sensitivity variations. WiSparse uses a weight-aware mechanism that combines activation magnitudes with precomputed weight norms to identify salient channels and employs a mixed-granularity allocation scheme optimized through evolutionary search. Experiments on Llama3.1 show that WiSparse, at 50% sparsity, preserves 97% of dense performance, outperforming the strongest baseline by 2.23 percentage points and achieving a 21.4% speedup in end-to-end inference.
LLMs can be sped up by 21% at inference time without retraining, thanks to a new sparsity method that smartly prunes activations based on the importance of the weights they interact with.
Large Language Models (LLMs) offer strong capabilities but incur high inference costs due to dense computation and memory access. Training-free activation sparsity is a promising approach for efficient LLM inference, yet existing methods often rely solely on activation information and uniform sparsity ratios. This overlooks the critical interplay with weights and inter-block sensitivity variation, leading to suboptimal performance. We identify two key phenomena in modern LLMs: 1) less significant activations may align with highly important weights, and 2) sparsity sensitivity varies non-monotonically across model blocks. We propose Weight-aware Mixed-Granularity Training-free Activation Sparsity (WiSparse), which leverages both activation and weight information for adaptive sparsity allocation. Specifically, we introduce a weight-aware mechanism integrating activation magnitudes with precomputed weight norms to accurately identify salient channels. This is combined with a mixed-granularity allocation scheme: a global budget is distributed across blocks via evolutionary search to protect sensitive regions, then refined within blocks to minimize reconstruction error. We improve sparse kernels and demonstrate effectiveness on three representative models. Notably, at 50% sparsity, WiSparse preserves 97% of Llama3.1's dense performance, surpassing the strongest baseline by 2.23 percentage points while achieving a 21.4% acceleration in end-to-end inference speed. Our research advances the limits of training-free approaches for efficient LLM inference, pushing the boundaries of achievable speedup without training.
Tsinghua AI