Search papers, labs, and topics across Lattice.
This paper introduces TOLiD, a self-supervised pretraining method that bridges the architecture gap between Vision Foundation Models (VFMs) and LiDAR backbones by utilizing token lifting for effective distillation. By coupling a LiDAR backbone with a Vision Transformer initialized from a frozen VFM teacher, the method enables compatibility between dense ViT token representations and sparse 3D encoders through Frustum Pooling and Frustum Attention. Extensive evaluations across five heterogeneous LiDAR datasets reveal significant improvements in transfer performance, even with frozen backbones and lightweight heads.
TOLiD achieves superior transfer learning in LiDAR representation by effectively bridging the modality and architecture gaps between vision and 3D data.
Cross-modal distillation from Vision Foundation Models (VFMs) to LiDAR backbones has recently emerged as a self-supervised pretraining strategy that reduces reliance on dense point-wise annotation for 3D scene understanding. However, existing distillation pipelines typically treat the VFM as a frozen feature source and train a heterogeneous 3D backbone to match fixed image embeddings, forcing the student to bridge both the modality gap and the cross-architecture gap between dense ViT token representations and sparse 3D encoders. We propose TOLiD, a self-supervised pretraining method for LiDAR representation learning that addresses this gap by coupling a LiDAR backbone with a student Vision Transformer (ViT) initialized from a frozen VFM teacher and applying supervision over compatible patch-token representations. TOLiD converts the set of point features within each image patch frustum into a token using Frustum Pooling followed by Frustum Attention, and performs token-level distillation with visibility masking. For LiDAR-only deployment, we lift token features back to per-point representations using masked bilinear sampling to avoid patches that have limited LiDAR points. We extensively evaluate TOLiD on five heterogeneous LiDAR datasets and four cross-sensor adaptation pairs, demonstrating improved transfer with frozen backbones and lightweight heads.
