Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper addresses the challenges of viewpoint invariance and temporal consistency in action detection within untrimmed videos. They introduce a two-stage approach: first, extracting motion features from augmented virtual viewpoints during training, and second, employing a novel view-invariant, multi-scale temporal encoder based on selective state-space sequence modelling. The proposed method, HydraView-TAD, achieves state-of-the-art performance on PKU-MMD and BABEL datasets, demonstrating significant improvements in action detection accuracy.
Synthesizing new viewpoints during training and encoding temporal dynamics with selective state-space models dramatically improves action detection, even when real-world videos have wildly different camera angles.
Viewpoint change invariance and action temporal consistency are critical aspects for the effective deployment of human action detection of untrimmed videos. Existing appearance-based video detection methods often struggle with limited viewpoint diversity during training, while motion-based detection approaches frequently fail to model fine-grained temporal relationships across consecutive motion windows. This paper introduces a novel two-stage action detection approach designed to improve both view-invariance and global temporal coherence properties. In the first stage, we extract motion features from augmented virtual viewpoints, solely used at training. Then, the second stage introduces a new view-invariant, multi-scale temporal encoder based on selective state-space sequence modelling to aggregate information across viewpoints and time scales. Experiments on PKU-MMD and BABEL benchmarks demonstrate that this approach significantly outperforms state-of-the-art methods in all considered splits. Code and trained models are available at: https://icb-vision-ai.github.io/HydraView-TAD