Search papers, labs, and topics across Lattice.
The paper addresses the challenge of transferring 3D pre-trained models to 4D point cloud video understanding tasks, which suffers from overfitting and modality gaps due to limited 4D data. To overcome these limitations, they propose a novel "Align then Adapt" (PointATA) paradigm that first aligns 3D and 4D feature distributions using optimal transport and then adapts the model with a point-video adapter and spatial-context encoder. Experiments demonstrate that PointATA achieves comparable or superior performance to full fine-tuning with significantly fewer parameters on 3D action recognition, 4D action segmentation, and 4D semantic segmentation tasks.
Bridging the gap between 3D and 4D point cloud understanding, PointATA unlocks surprisingly strong performance with parameter-efficient transfer learning, even surpassing full fine-tuning.
Point cloud video understanding is critical for robotics as it accurately encodes motion and scene interaction. We recognize that 4D datasets are far scarcer than 3D ones, which hampers the scalability of self-supervised 4D models. A promising alternative is to transfer 3D pre-trained models to 4D perception tasks. However, rigorous empirical analysis reveals two critical limitations that impede transfer capability: overfitting and the modality gap. To overcome these challenges, we develop a novel"Align then Adapt"(PointATA) paradigm that decomposes parameter-efficient transfer learning into two sequential stages. Optimal-transport theory is employed to quantify the distributional discrepancy between 3D and 4D datasets, enabling our proposed point align embedder to be trained in Stage 1 to alleviate the underlying modality gap. To mitigate overfitting, an efficient point-video adapter and a spatial-context encoder are integrated into the frozen 3D backbone to enhance temporal modeling capacity in Stage 2. Notably, with the above engineering-oriented designs, PointATA enables a pre-trained 3D model without temporal knowledge to reason about dynamic video content at a smaller parameter cost compared to previous work. Extensive experiments show that PointATA can match or even outperform strong full fine-tuning models, whilst enjoying the advantage of parameter efficiency, e.g. 97.21 \% accuracy on 3D action recognition, $+8.7 \%$ on 4 D action segmentation, and 84.06\% on 4D semantic segmentation.
