Search papers, labs, and topics across Lattice.
This paper introduces RbFT-Net, a novel framework for enhancing depth completion from sparse radar and camera data by implementing a rectify-before-fuse strategy. The method addresses challenges such as temporal misalignment and measurement noise by correcting radar measurements before they are fused with camera data, thus improving the reliability of depth predictions. Experimental results demonstrate that RbFT-Net outperforms existing independent methods and competes effectively with more complex pipelines, validating its approach to managing unreliable sensor data.
RbFT-Net reveals that correcting radar measurements before fusion can significantly enhance depth completion accuracy in autonomous systems.
Dense metric depth prediction from cameras and millimeter-wave radar offers a cost-effective sensing solution for autonomous systems. However, radar measurements are inherently sparse and susceptible to clutter, multipath reflections, and projection errors. While aggregating multiple radar frames provides denser metric cues, it also introduces temporal misalignment and dynamic-object interference. Directly propagating such unreliable measurements can therefore corrupt large regions of the predicted depth map. To address this issue, we propose RbFT-Net, an end-to-end rectify-before-fuse framework for multi-frame 4D radar-camera depth completion. Rather than assuming accumulated radar returns to be accurate, RbFT-Net treats them as noisy temporal anchor candidates. An image-conditioned rectification module jointly corrects their image-plane locations and metric depths while estimating pointwise reliability. The rectified anchors are then selectively propagated before high-level multi-modal fusion, suppressing the influence of unreliable measurements. Experiments on ZJU-4DRadarCam and a newly collected 4D radar-camera-LiDAR dataset show that RbFT-Net consistently outperforms the evaluated independent radar-camera methods and remains competitive with plug-in pipelines using auxiliary monocular depth models. Cross-platform evaluation and component analyses further support the effectiveness of the proposed rectification and reliability-aware propagation strategy.
