Search papers, labs, and topics across Lattice.
This paper introduces CRISP, a novel spatiotemporal backbone for camera-radar fusion in autonomous driving, which leverages forecasting-based representation learning to create a unified bird's-eye-view (BEV) representation from historical multi-view images and radar sweeps. By using LiDAR solely for privileged supervision during pretraining, CRISP enables the deployment of models that rely only on camera and radar inputs, enhancing their practical applicability. Experimental results on the nuScenes dataset demonstrate significant improvements in long-horizon point cloud forecasting and effective transfer to various downstream tasks, highlighting the potential of predictive CR pretraining for scalable driving representations.
CRISP achieves superior long-horizon point cloud forecasting and versatile downstream task performance by leveraging a unique forecasting-based pretraining approach with camera-radar fusion.
Camera-radar (CR) fusion is a practical sensing configuration for autonomous driving, but existing models are typically trained with task-specific supervision, limiting reusable representation learning. We present CRISP, a spatiotemporal CR backbone pretrained through forecasting-based representation learning. Given historical multi-view images and radar sweeps, CRISP learns a unified bird's-eye-view (BEV) representation by predicting future LiDAR point clouds. LiDAR is used only as privileged supervision during pretraining; the deployed model requires only camera and radar. To make forecasting-based pretraining effective for CR fusion, CRISP introduces an enhanced radar encoder, radar-enhanced temporal self-attention, and multimodal feature rendering with modality innovation gating. These components inject radar range and Doppler cues into BEV temporal propagation and allow BEV tokens to selectively incorporate camera and radar evidence. Experiments on nuScenes show that CRISP improves long-horizon point cloud forecasting and transfers effectively to downstream tasks, including 3D detection, tracking, online mapping, motion forecasting, future occupancy prediction, and planning, suggesting that predictive CR pretraining is a promising path toward scalable driving representations under practical sensor configurations. The project website is https://umfieldrobotics.github.io/CRISP.
