Search papers, labs, and topics across Lattice.
This paper introduces Prox, a two-stage training-free framework designed to enhance activation sparsity in feed-forward networks (FFNs) of large language models (LLMs) by leveraging the SwiGLU intermediate state for effective channel selection. By constructing a channel mask from the magnitude ranking of intermediate state entries rather than their exact values, Prox significantly mitigates the model-quality degradation typically associated with high sparsity levels. The results demonstrate that Prox achieves superior performance compared to existing training-free methods, yielding up to a 1.99x speedup in end-to-end decoding at 70% FFN sparsity across ten LLMs from six model families.
Sparsifying FFNs can lead to nearly double the decoding speed without sacrificing model quality, thanks to a novel channel-selection strategy.
Feed-forward networks (FFNs) dominate memory traffic and computation in large language model (LLM) inference, making them a primary target for activation sparsification. However, existing training-free methods suffer substantial model-quality degradation at high sparsity due to limitations in their channel-selection strategies. We observe that the SwiGLU intermediate state provides a highly effective channel-selection signal, but obtaining it requires costly dense computation. To address this, we present \emph{Prox}, a two-stage training-free framework for sparse SwiGLU FFNs. Prox hinges on the key insight: sparse execution requires only the channel mask induced by the intermediate state, which can be constructed from the magnitude ranking of its entries rather than their exact values. Specifically, Stage 1 uses input sparsity and quantized proxy weights to construct a shared mask; Stage 2 computes the selected channels exactly, enabling sparse execution of all three projections. Across ten LLMs from six model families, Prox outperforms training-free baselines at all sparsity levels, achieves up to a $1.99\times$ end-to-end decoding speedup at 70\% FFN sparsity, and is compatible with quantization and sparse attention.
