Search papers, labs, and topics across Lattice.
This paper introduces SpikeDecoder, a fully spiking neural network (SNN) implementation of the Transformer decoder block aimed at natural language processing tasks. By analyzing the trade-offs of integrating SNNs with various model components and exploring different embedding methods for text data, the authors achieve a significant reduction in energy consumption. The results indicate that SpikeDecoder can lower theoretical energy usage by 87% to 93% compared to traditional artificial neural networks, highlighting its potential for energy-efficient NLP applications.
Energy consumption in NLP could plummet by up to 93% with the SpikeDecoder, a fully spiking neural network implementation of the Transformer decoder.
The Transformer architecture is widely regarded as the most powerful tool for natural language processing, but due to a high number of complex operations, it inherently faces the issue of high energy consumption. To address this issue, we consider Spiking Neural Networks (SNNs), which are an energy-efficient alternative to conventional Artificial Neural Networks (ANNs) due to their naturally event-driven approach to processing information. However, this inherently makes them difficult to train. Often, many SNN-based models circumvent this issue by converting pre-trained ANNs. More recently, attempts have been made to design directly trainable SNN-based adaptations of the Transformer model structure. Although the results showed great promise, the application field was computer vision. Moreover, the proposed model incorporates only encoder blocks. In this paper, we propose SpikeDecoder, a fully SNN-based implementation of the Transformer decoder block, for applications in natural language processing. In a series of experiments, we analyze the impact of exchanging different blocks of the ANN model with spike-based alternatives to identify trade-offs and significant sources of performance loss. We further investigate the role of residual connections and the selection of SNN-compatible normalization techniques. Besides the work on the model architecture, we formulate and compare different embedding methods to project text data into spikes. Finally, we demonstrate that our proposed SNN-based decoder block reduces the theoretical energy consumption by 87% to 93% compared to the ANN baseline.
