Search papers, labs, and topics across Lattice.
FOMO-3D is introduced as a novel multi-modal 3D object detector that leverages vision foundation models (OWLv2 and Metric3Dv2) to address the challenge of long-tailed 3D detection in autonomous driving. The architecture employs a two-stage approach, generating proposals from LiDAR and camera data, and refining them using attention mechanisms focused on image features from OWL. Experiments on real-world driving datasets demonstrate significant improvements in detecting rare, safety-critical objects by incorporating semantic and depth priors from vision foundation models.
Vision foundation models can be effectively adapted to significantly improve the detection of rare, safety-critical objects in 3D autonomous driving scenarios, even with limited training data.
In order to navigate complex traffic environments, self-driving vehicles must recognize many semantic classes pertaining to vulnerable road users or traffic control devices. However, many safety-critical objects (e.g., construction worker) appear infrequently in nominal traffic conditions, leading to a severe shortage of training examples from driving data alone. Recent vision foundation models, which are trained on a large corpus of data, can serve as a good source of external prior knowledge to improve generalization. We propose FOMO-3D, the first multi-modal 3D detector to leverage vision foundation models for long-tailed 3D detection. Specifically, FOMO-3D exploits rich semantic and depth priors from OWLv2 and Metric3Dv2 within a two-stage detection paradigm that first generates proposals with a LiDAR-based branch and a novel camera-based branch, and refines them with attention especially to image features from OWL. Evaluations on real-world driving data show that using rich priors from vision foundation models with careful multi-modal fusion designs leads to large gains for long-tailed 3D detection. Project website is at https://waabi.ai/fomo3d/.
