Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces a novel video transformer architecture that incorporates a sparse winner-takes-all token selection mechanism, enhancing efficiency by mimicking competitive routing in biological visual systems. By utilizing a dual-stream approach—combining a high-resolution "what" stream with a low-resolution "where" stream—the model achieves superior performance on Kinetics-400 and Something-Something V2, operating at the Pareto frontier of accuracy and inference time. The model demonstrates a peak correlation of 0.18 with EEG data, indicating that these neuro-inspired design choices significantly improve both robustness and biological interpretability in video understanding tasks.
Sparse competition and dual-stream processing lead to a video transformer that not only matches state-of-the-art accuracy but also aligns closely with brain activity.
Modern video transformers typically ignore principles from primate vision and are rarely evaluated against neural data, limiting their biological interpretability. We introduce a sparse winner-takes-all token selection module that replaces dense self-attention to improve efficiency and approximate competitive routing observed in biological visual circuits. We further propose a neuro-inspired split-and-fuse video transformer which uses two complementary pathways: a high-resolution, low-frame-rate "what" stream and a low-resolution, high-frame-rate "where" stream, fused before classification. On Kinetics-400 and Something-Something V2, our best variant operates on the Pareto frontier of accuracy versus inference time among models of comparable scale and pretraining, and showing improved robustness to spatial perturbations. Using representational similarity analysis between model embeddings and time-resolved EEG recordings for the same video stimuli, our model attains a peak brain-model correlation of 0.18 (about 78% of the noise ceiling) and consistently outperforms strong video transformer baselines, suggesting that pathway specialization and sparse competition are useful inductive biases for efficient, brain-aligned video understanding.