Search papers, labs, and topics across Lattice.
This paper introduces SpikeTAD, the first end-to-end temporal action detection (TAD) architecture based on Spiking Neural Networks (SNNs), addressing the challenges of large model sizes and high power consumption in video understanding tasks. By leveraging the bioplausibility and efficiency of SNNs, SpikeTAD achieves competitive performance with an average mean Average Precision (mAP) of 67.2% on the THUMOS14 dataset and 37.42% on ActivityNet-1.3, while maintaining significantly lower power requirements. This work highlights the potential of SNNs for practical deployment in mobile devices, paving the way for more efficient video understanding systems.
Achieving a competitive 67.2% mAP in temporal action detection with a spiking neural network model that consumes significantly less power than traditional approaches.
Video understanding is a crucial part of computer vision, with numerous application scenarios. With the increasing popularity of mobile devices, an increasing number of efforts are trying to deploy video understanding models on them. However, existing video understanding models are difficult to deploy due to their large size and prohibitive power consumption. Spiking Neural Networks (SNNs) have shown bioplausibility and low power advantages over Artificial Neural Networks (ANNs), especially on neuromorphic chips which are regarded as essential components of future mobile devices. However, excessively long conversion time-steps and severe performance degradation problems limit their application. To solve the problems above, we explore the application of SNNs on temporal action detection (TAD), which is an important task in video understanding, and propose the first SNN-based end-to-end TAD architecture coined as SpikeTAD. While maintaining extremely low power consumption, SpikeTAD achieves an average mAP of 67.2% in THUMOS14 and 37.42% in ActivityNet-1.3, demonstrating the feasibility of a low-power TAD model. Our code is available at https://github.com/MCG-NJU/SpikeTAD.
