Search papers, labs, and topics across Lattice.
This paper introduces FlexibleFusion, an adaptive method for infrared-visible object detection (IVOD) that effectively manages the integration of visible and infrared data, even when one modality is missing. The method employs a Modality-Aware Experts Collaboration (MAEC) mechanism to dynamically activate cross-modal or intra-modal pathways based on the availability of sensor data, ensuring robust performance in challenging conditions. Additionally, the Residual Self-Paced Entropic Optimal Transport (RSPEOT) approach enhances feature alignment by refining matches progressively, leading to consistent detection accuracy across various modality configurations.
FlexibleFusion's innovative approach allows for seamless object detection even when critical sensor data is missing, transforming how we handle multimodal perception in real-world applications.
Infrared-visible object detection (IVOD) integrates complementary evidence from visible and infrared sensors for reliable perception in challenging scenes. In practice, sensors may fail or drop frames, leaving one modality unavailable or intermittent. Existing methods for IVOD assume both modalities are always present, and fixed fusion collapses when one stream is missing. Furthermore, it remains a critical challenge to reliably estimate semantic correlation across heterogeneous modalities, especially under spectral distribution discrepancy. We present FlexibleFusion, a unified and adaptive method that flexibly allocates integration pathways and fusion strength, operating seamlessly across complete and missing-modality regimes. At its core, the Modality-Aware Experts Collaboration (MAEC) mechanism selectively activates and aggregates cross-modal or intra-modal expert pathways. It allows cross-modal fusion when full modalities are available and falls back to self-fusion under missing conditions. Additionally, we design Residual Self-Paced Entropic Optimal Transport (RSPEOT) to align heterogeneous feature distributions from a transport perspective. Instead of relying on the fixed sparsity coefficient in standard entropic optimal transport (EOT), RSPEOT introduces a residual-driven self-paced update that prioritizes reliable matches and progressively refines harder ones. This design alleviates the additional optimization burden of standard EOT while preserving reliable semantic alignment. Comprehensive experiments under complete and missing-modality protocols show consistent performance across arbitrary modality configurations. Code will be released upon publication.
