Search papers, labs, and topics across Lattice.
This paper introduces ViCo3D, a novel framework that enhances LiDAR-based collaborative 3D object detection by leveraging vision foundation models (VFMs) to bridge the gap between point cloud and image modalities. By projecting point clouds onto the bird's-eye-view (BEV) plane and employing a multi-scale BEV fusion module, ViCo3D effectively integrates visual features with geometric data, significantly improving collaboration among agents. Experimental results on DAIR-V2X and V2XSet show that ViCo3D achieves state-of-the-art performance, with up to 1.8 times greater collaborative gains compared to existing methods.
ViCo3D achieves a groundbreaking 1.8x improvement in collaborative 3D object detection performance by merging LiDAR data with advanced visual representations from vision foundation models.
LiDAR-based collaborative 3D perception in Vehicle-to-Everything (V2X) systems typically relies on fusing bird's-eye-view (BEV) features across agents. However, current BEV representations, typically extracted by LiDAR backbones trained from scratch, are geometry-dominated and lack general semantic priors, inherently limiting the efficacy of feature-level collaboration. Meanwhile, vision foundation models (VFMs) pretrained on large-scale image data have demonstrated strong capability in learning general-purpose and informative visual representations for 2D tasks, and have the potential to enhance agent-wise LiDAR BEV representations for collaboration. Despite this potential, adapting VFMs to LiDAR-based 3D detection remains challenging due to the substantial image-point cloud modality gap. To bridge this gap, we propose ViCo3D, a collaborative 3D object detection framework powered by VFMs. Specifically, ViCo3D adapts VFMs to LiDAR-based collaborative perception from three aspects: First, ViCo3D projects point clouds onto the BEV plane as three-channel images, enabling DINOv2 to extract BEV-space visual features from LiDAR inputs. Besides, to effectively integrate these DINOv2-derived features with LiDAR geometric features, ViCo3D introduces a multi-scale BEV fusion module within the single-agent encoder. In addition, ViCo3D adopts an ego-centric cross-agent fusion strategy to aggregate complementary information from multiple agents. Experiments on DAIR-V2X and V2XSet demonstrate that ViCo3D achieves state-of-the-art 3D detection performance. Remarkably, it delivers up to 1.8x greater collaborative gains than prior methods on DAIR-V2X. The code will be made public available for future investigation.
