Search papers, labs, and topics across Lattice.
This paper introduces Multi-Order Self-Similarity (MOSS), a neural module that learns and integrates space-time self-similarity (STSS) features of varying orders to capture distinct aspects of video dynamics. By explicitly modeling higher-order STSS, the module enhances motion modeling capabilities for video understanding tasks. Experiments across action recognition, video VQA, and robotic tasks show that MOSS provides substantial performance gains with minimal computational overhead.
Modeling higher-order relationships between video frames unlocks surprisingly large gains in action recognition, VQA, and robotics, all with a lightweight module.
Space-time self-similarity (STSS), which captures visual correspondences across frames, provides an effective way to represent temporal dynamics for video understanding. In this work, we explore higher-order STSS and demonstrate how STSSs at different orders reveal distinct aspects of these dynamics. We then introduce the Multi-Order Self-Similarity (MOSS) module, a lightweight neural module designed to learn and integrate multi-order STSS features. It can be applied to diverse video tasks to enhance motion modeling capabilities while consuming only marginal computational cost and memory usage. Extensive experiments on video action recognition, motion-centric video VQA, and real-world robotic tasks consistently demonstrate substantial improvements, validating the broad applicability of MOSS as a general temporal modeling module. The source code and checkpoints will be publicly available.
