Track manipulation, locomotion and embodied systems.
Search papers, labs, and topics across Lattice.
Lang3DSeg introduces a point transformer architecture for annotation-free, open-vocabulary 3D LiDAR segmentation in outdoor environments, overcoming the limitations of previous voxel-based methods and indoor-only point transformers. The approach addresses projection noise from 2D vision-language models by compositing masks with a class-priority rule and truncating instances based on depth distribution gaps. This method achieves state-of-the-art performance on nuScenes and SemanticKITTI without geometric pre-training or real-time vision-language model inference.
Open-vocabulary 3D LiDAR segmentation now leverages point transformers for outdoor scenes, achieving SOTA without manual annotation or geometric pre-training.
Accurate 3D semantic perception is critical for safe autonomous navigation. However, supervised LiDAR segmentation remains tied to closed taxonomies and to the cost of point-wise manual annotation. Open-vocabulary methods avoid that cost by projecting the output of 2D vision-language models onto LiDAR and distilling it into a 3D network. These methods rely almost exclusively on voxel-based sparse convolutions, and point transformers have so far been limited to indoor environments, where 3D data is dense and bounded. We present Lang3DSeg, which establishes a point transformer as the backbone for annotation-free open-vocabulary segmentation of outdoor 3D LiDAR, and is trained from scratch without geometric pre-training. This training paradigm necessitates addressing the inherent noise in 2D-to-3D label projections; specifically, naive projection often suffers from depth ambiguity, where points behind an object are erroneously assigned its semantic label. We therefore composite masks using an explicit class-priority rule and truncate each projected instance at the first gap in its depth distribution, correcting the projection error directly rather than averaging it over registered sequences. Lang3DSeg achieves 52.8% mIoU on nuScenes validation and 41.4% on SemanticKITTI, the highest among published annotation-free methods on both benchmarks. Every 3D semantic segmentation is on a single LiDAR sweep, and inference operates in real-time without running vision-language models.