Search papers, labs, and topics across Lattice.
This paper introduces the SMM Transformer, a novel framework that integrates Spiking Neural Networks (SNNs) for multimodal tasks by addressing the challenges of unstable training and the inefficiencies of traditional attention mechanisms. The framework employs a Parallel LIF neuron model and a tailored P-STBP algorithm for stable training, alongside a spike-driven token-mixing module that significantly reduces computational energy consumption. Experimental results demonstrate that SMM Transformer achieves competitive accuracy with artificial neural network (ANN) baselines while offering substantial energy efficiency improvements, particularly in the attention module.
Achieving up to 97% reduction in energy consumption for attention mechanisms, the SMM Transformer redefines efficiency in multimodal SNN applications.
Spiking Neural Networks (SNNs) enable event-driven computation with sparse activations, but building multimodal Transformers on SNNs is hindered by unstable training in deep spiking stacks and the mismatch between dense softmax attention and spike-based communication. We propose SMM Transformer, an SNN-based multimodal Transformer framework that combines (i)PLMP, a Parallel LIF with Multistage Learnable Parameters neuron and a tailored P-STBP algorithm for stable deep SNN training, (ii) SMSA, an attention-inspired spike-driven token-mixing module that replaces dense pairwise softmax attention with channel-wise spike co-activation and self-compensation, and (iii)SMoE, a spiking mixture-of-experts module for modality-aware fusion. Across visual and multimodal benchmarks, SMM Transformer achieves competitive accuracy compared to ANN baselines. Under a standard MAC/AC arithmetic model, SMSA reduces the estimated operator-level compute energy of the attention module by up to 97%, while whole-model profiling shows more moderate but consistent efficiency gains.
