Search papers, labs, and topics across Lattice.
This paper introduces a method for inducing sparse neural activity in quantized linear-attention models, effectively reducing computational costs while maintaining performance. By nullifying activations below a trainable threshold, the approach achieves comparable performance to dense models with significantly fewer arithmetic operations. The results indicate that this method can enhance throughput and reduce power consumption on neuromorphic hardware, projecting up to 37脳 higher throughput compared to traditional GPU architectures.
Sparse, quantized linear-attention models can outperform dense counterparts on neuromorphic hardware, achieving up to 37脳 higher throughput and 16脳 lower power consumption.
Inference with transformer-based large language models (LLMs) is often limited by the memory-bound KV cache and quadratic attention cost. State-space models (SSMs) mitigate this through linear attention and fixed-size recurrent states, but their large dense linear projections remain computationally expensive even after quantization. We introduce a method that induces sparse neural activity in heavily quantized linear-attention models with minimal performance loss. Activations below a per-projection trainable threshold ($\pm 螖$) are nullified while preserving crucial outliers, achieving comparable performance to dense models with up to 4$\times$ fewer effective arithmetic operations. Targeting a multi-core, multi-chip neuromorphic platform, where event-driven execution converts unstructured sparsity into throughput at both the compute and communication levels, a capability GPU architectures fundamentally lack, we project up to 37$\times$ higher throughput and 16$\times$ lower power versus edge GPU inference of a comparable transformer-based model, and up to 5.4$\times$ improvements over the non-sparsified baseline. These results position sparse, quantized linear-attention models as a natural fit for deploying LLMs on event-driven multi-core platforms.
