Search papers, labs, and topics across Lattice.
The paper introduces STS-Mixer, a novel framework for 4D point cloud video understanding that leverages a spectral perspective to capture geometric characteristics. It transforms 4D point cloud videos into graph spectral signals, decomposing them into frequency bands that represent coarse shapes (low-frequency) and fine-grained details (high-frequency). STS-Mixer then integrates these multi-band spectral signals with spatiotemporal information, achieving state-of-the-art performance on 3D action recognition and 4D semantic segmentation tasks.
Decomposing 4D point cloud videos into spectral frequency bands unlocks superior geometric understanding, boosting performance on action recognition and semantic segmentation.
4D point cloud videos capture rich spatial and temporal dynamics of scenes which possess unique values in various 4D understanding tasks. However, most existing methods work in the spatiotemporal domain where the underlying geometric characteristics of 4D point cloud videos are hard to capture, leading to degraded representation learning and understanding of 4D point cloud videos. We address the above challenge from a complementary spectral perspective. By transforming 4D point cloud videos into graph spectral signals, we can decompose them into multiple frequency bands each of which captures distinct geometric structures of point cloud videos. Our spectral analysis reveals that the decomposed low-frequency signals capture more coarse shapes while high-frequency signals encode more fine-grained geometry details. Building on these observations, we design Spatio-Temporal-Spectral Mixer (STS-Mixer), a unified framework that mixes spatial, temporal, and spectral representations of point cloud videos. STS-Mixer integrates multi-band delineated spectral signals with spatiotemporal information to capture rich geometries and temporal dynamics, while enabling fine-grained and holistic understanding of 4D point cloud videos. Extensive experiments show that STS-Mixer achieves superior performance consistently across multiple widely adopted benchmarks on both 3D action recognition and 4D semantic segmentation tasks. Code and models are available at https://github.com/Vegetebird/STS-Mixer.
DAMO