Search papers, labs, and topics across Lattice.
This paper introduces RSFusionDet, a novel RGB-Sonar multimodal object detection framework that leverages the complementary strengths of optical and sonar imaging to enhance underwater object detection. By creating the RSFusion dataset and implementing a Cross-Attention Fusion (CAFusion) module, the authors address the challenges of spatial misalignment and object matching across modalities. The proposed method achieves state-of-the-art performance with 76.4/48.6 AP for RGB and Sonar detection, respectively, and an 83.4 F1-Score for object matching, surpassing existing models including the DINO baseline.
RSFusionDet not only outperforms existing models in underwater object detection but also achieves reliable cross-modal object matching, setting a new benchmark in the field.
Underwater unimodal object detection faces many challenges in sensor imaging, such as optical images limited by underwater noise and visible distance, and sonar images limited by less object structural information. While, optical images have rich object structural information, and sonar images are less affected by underwater noise and have a longer visible distance. Optical (RGB modality) and sonar (Sonar modality) images have complementary information underwater. In this paper, we create an RGB-Sonar multimodal object detection dataset, \textbf{R}GB-\textbf{S}onar \textbf{Fusion} (RSFusion) and propose evaluation metrics for the benchmark. And we propose the \textbf{R}GB-\textbf{S}onar \textbf{Fusion} \textbf{Det}ector (RSFusionDet) with a new RGB-Sonar multimodal object detection result expression for RGB-Sonar multimodal object detection. We analyze the features of RGB and Sonar modal information, and design a Cross-Attention Fusion (CAFusion) module to fuse RGB-Sonar spatial misalignment features and Object Matching Head (OMHead) with Loss (OMLoss) to match identical objects in RGB-Sonar modalities. Our RSFusionDet achieves 76.4/48.6 AP (RGB/Sonar) for object detection and 83.4 \(\text{F1-Score}_{match}\) for object matching, on RSFusion, which outperforms other object detection models. Compared with the DINO baseline, our method improves by 0.7/1.4 AP (RGB/Sonar) while simultaneously providing reliable cross-modal object matching. The code and datasets are publicly available at https://github.com/LEFTeyex/RSFusionDet.
