Search papers, labs, and topics across Lattice.
The paper introduces the Overall Glance and Refined Gaze (OG-ReG) Transformer, a dual-path network designed to mimic human visual attention for improved video understanding. The model uses a "Glance" path for coarse spatiotemporal information and a "Gaze" path for local details, addressing limitations of factorized attention mechanisms in capturing long-range dependencies. OG-ReG achieves state-of-the-art results on Kinetics-400, Something-Something v2, and Diving-48, demonstrating the effectiveness of mimicking human visual attention in video transformers.
Mimicking the human visual system's "glance and gaze" strategy lets video transformers capture long-range dependencies and achieve state-of-the-art action recognition.
Recently, Transformer has made significant progress in various vision tasks. To balance computation and efficiency in video tasks, recent works heavily rely on factorized or window-based self-attention. However, these approaches split spatiotemporal correlations between regions of interest in videos, limiting the models'ability to capture motion and long-range dependencies. In this paper, we argue that, similar to the human visual system, the importance of temporal and spatial information varies across different time scales, and attention is allocated sparsely over time through glance and gaze behavior. Is equal consideration of time and space crucial for success in video tasks? Motivated by this understanding, we propose a dual-path network called the Overall Glance and Refined Gaze (OG-ReG) Transformer. The Glance path extracts coarse-grained overall spatiotemporal information, while the Gaze path supplements the Glance path by providing local details. Our model achieves state-of-the-art results on the Kinetics-400, Something-Something v2, and Diving-48, demonstrating its competitive performance. The code will be available at https://github.com/linuxsino/OG-ReG.
