Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper explores the feasibility of distilling semantic segmentation knowledge from Visual Foundation Models (VFMs) to lidar point clouds for indoor scenes, addressing the scarcity of labeled lidar data. They create pseudo-labels by projecting VFM-processed camera images onto lidar scans and use these to train a lidar segmentation model. Experiments on indoor SLAM datasets demonstrate that the distilled model achieves a reasonable mIoU (up to 56% with pseudo-labels and 36% with real labels), indicating the potential of this cross-modal distillation approach.
Skip the costly lidar annotations: VFMs can be distilled to semantically segment indoor lidar scans with reasonable accuracy.
Frame-wise semantic segmentation of indoor lidar scans is a fundamental step toward higher-level 3D scene understanding and mapping applications. However, acquiring frame-wise ground truth for training deep learning models is costly and time-consuming. This challenge is largely addressed, for imagery, by Visual Foundation Models (VFMs) which segment image frames. The same VFMs may be used to train a lidar scan frame segmentation model via a 2D-to-3D distillation pipeline. The success of such distillation has been shown for autonomous driving scenes, but not yet for indoor scenes. Here, we study the feasibility of repeating this success for indoor scenes, in a frame-wise distillation manner by coupling each lidar scan with a VFM-processed camera image. The evaluation is done using indoor SLAM datasets, where pseudo-labels are used for downstream evaluation. Also, a small manually annotated lidar dataset is provided for validation, as there are no other lidar frame-wise indoor datasets with semantics. Results show that the distilled model achieves up to 56% mIoU under pseudo-label evaluation and around 36% mIoU with real-label, demonstrating the feasibility of cross-modal distillation for indoor lidar semantic segmentation without manual annotations.