Search papers, labs, and topics across Lattice.
The paper introduces Single-Modality-Operable Multimodal Collaborative Perception (SiMO) to address the vulnerability of existing multimodal collaborative perception systems to sensor failures. SiMO uses Length-Adaptive Multi-Modal Fusion (LAMMA) to adaptively handle remaining modal features during failures and a "Pretrain-Align-Fuse-RD" training strategy to ensure the independence of each modality branch. Experiments show SiMO aligns multimodal features while preserving modality-specific features, maintaining optimal performance across individual modalities even when some sensors fail.
Collaborative perception that gracefully handles sensor failures, maintaining performance even when key modalities like LiDAR are unavailable, by ensuring semantic consistency across modalities.
Collaborative perception integrates multi-agent perspectives to enhance the sensing range and overcome occlusion issues. While existing multimodal approaches leverage complementary sensors to improve performance, they are highly prone to failure--especially when a key sensor like LiDAR is unavailable. The root cause is that feature fusion leads to semantic mismatches between single-modality features and the downstream modules. This paper addresses this challenge for the first time in the field of collaborative perception, introducing Single-Modality-Operable Multimodal Collaborative Perception (SiMO). By adopting the proposed Length-Adaptive Multi-Modal Fusion (LAMMA), SiMO can adaptively handle remaining modal features during modal failures while maintaining consistency of the semantic space. Additionally, leveraging the innovative "Pretrain-Align-Fuse-RD" training strategy, SiMO addresses the issue of modality competition--generally overlooked by existing methods--ensuring the independence of each individual modality branch. Experiments demonstrate that SiMO effectively aligns multimodal features while simultaneously preserving modality-specific features, enabling it to maintain optimal performance across all individual modalities. The implementation details can be found in https://github.com/dempsey-wen/SiMO.
