Search papers, labs, and topics across Lattice.
This paper introduces MATS, a multi-modality multi-task learning framework designed to enhance 3D perception in autonomous driving by integrating diverse sensor data more effectively. By employing a modality-adaptive BEV fusion module and a task-specific Mixture-of-Experts (MoE) approach, MATS generates tailored BEV feature maps that significantly improve performance across various perception tasks. Extensive experiments on the nuScenes benchmark demonstrate that MATS outperforms state-of-the-art methods, showcasing its potential to advance the field of autonomous driving perception systems.
MATS achieves a substantial performance boost in 3D perception tasks by intelligently adapting BEV feature maps for specific modalities and tasks.
Multi-modality data from different sensors provides rich complementary information for 3D perception, becoming an essential component in reliable autonomous driving systems. Current research typically designs intricate and complex fusion strategies to integrate information from multimodal data on a unified bird's-eye-view (BEV) feature map for the joint learning of multiple perception tasks. However, such a single feature map hardly carries sufficient information to simultaneously meet the requirements of various perception tasks, leading to a very limited perception performance. To mitigate this limitation, this paper proposes MATS, a novel multi-modality multi-task learning approach with modality-adaptive BEV fusion and task-specific Mixture-of-Experts (MoE) for 3D perception. Specifically, a simple modality-adaptive BEV fusion module is designed to adaptively recalibrate the BEV features by modeling the global cross-modality dependencies, generating diverse BEV feature maps for various perception tasks. For joint multi-task learning, this paper proposes a task-specific MoE module to decouple the tasks and enable the network to automatically choose the appropriate BEV feature candidates for each specific task. To validate the effectiveness of the proposed approach, we conduct extensive experiments on the large-scale benchmark nuScenes. With the camera- and LiDAR-modality input data, the proposed approach outperforms the state-of-the-art (SOTA) by a significant margin. Furthermore, the experimental results on the single tasks show that the proposed approach significantly outperforms the baselines. The code and trained models will be available upon publication.
