Search papers, labs, and topics across Lattice.
This paper introduces VideoSEMA, a novel split space-time attention model designed for efficient video classification, which combines a scalable Mamba-like attention block for spatial processing with softmax temporal attention. The authors demonstrate that under specific rank conditions, this computationally efficient model can achieve performance comparable to full space-time attention while significantly outperforming heavier vision transformer and Mamba models on benchmark datasets like K400 and SSv2. Notably, VideoSEMA exhibits superior accuracy retention as image resolution increases, suggesting its robustness for longer video sequences without requiring fine-tuning.
VideoSEMA outperforms heavier models while maintaining efficiency, achieving top accuracy even as image resolution scales up.
We present for video understanding (classification) a split space-time attention model, VideoSEMA, consisting of a scalable and efficient Mamba-like attention (SEMA) block in space and a softmax temporal attention in time. In each frame, SEMA attention applies a local window attention in parallel with a global averaging in a Mamba macro-architecture, which is called Mamba-like. Under certain rank conditions, we prove that the computationally cheaper split space-time attention is equivalent to full space-time attention. On benchmark K400 data sets, VideoSEMA out-performs heavier vision transformer and Mamba models. On benchmark SSv2 data, VideoSEMA leads in top-1 accuracy among models of similar parameter sizes. As image resolution scales up from standard $224^2$ to $1024^2$ on K400 and without fine-tuning, VideoSEMA degrades much more gracefully than VideoMamba in accuracy. It is promising to extend VideoSEMA to longer videos with a dilated/sparse temporal attention.
