Search papers, labs, and topics across Lattice.
This paper introduces open-vocabulary BEV segmentation (OVBS) to enhance bird's-eye view perception for autonomous driving by enabling recognition of categories beyond the training set while ensuring real-time efficiency. The authors tackle the challenge of 3D geometric inconsistency in translating 2D vision-language model semantics into BEV through a novel framework called OVBEVSeg, which employs a three-stage process involving pseudo-labeling, joint optimization, and geometric distillation. On the nuScenes dataset, OVBEVSeg achieves state-of-the-art performance, surpassing closed-set methods by 15.3 mIoU on unseen categories and demonstrating significant improvements in inference speed and memory efficiency.
Achieving 15.3 mIoU improvement on unseen categories, OVBEVSeg redefines BEV segmentation by enabling open-vocabulary recognition with real-time efficiency.
Bird's-eye view (BEV) perception fuses multi-camera images into a unified top-down representation for autonomous driving. Despite recent progress, state-of-the-art methods remain confined to closed-set scenarios, making them vulnerable to unpredictable real-world environments. In this work, we introduce open-vocabulary BEV segmentation (OVBS), which leverages vision-language models (VLMs) to recognize categories beyond the training set while maintaining precise BEV perception and real-time efficiency. A key challenge in OVBS lies in the 3D geometric inconsistency inherent in the ill-posed lifting of 2D VLM semantics into BEV. To address this, we propose OVBEVSeg, a geometry-aware OVBS framework that enhances efficient Gaussian splatting (GS)-based unprojection by leveraging robust 3D geometric constraints across three progressive stages: (1) 2D-to-BEV pseudo-labeling via reliable 3D projection for OV generalization; (2) joint 2D-BEV per-scene optimization with BEV structural constraints for 3D geometric consistency; and (3) 3D geometric distillation for online efficiency. On the nuScenes dataset, OVBEVSeg achieves state-of-the-art performance, outperforming closed-set methods by 15.3 mIoU on unseen categories. Remarkably, even with no novel-class ground-truth labels, it remains competitive with self- and semi-supervised baselines trained with up to 40% of ground-truth annotations. Furthermore, it achieves 2.5x faster inference with only 0.22x the memory consumption of projection-based methods. Project page: https://hchoi256.github.io/projects/ovbevseg/.
