Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces SonarLLM, a multimodal large language model that integrates sonar as a native perceptual modality alongside optical inputs to enhance underwater perception under varying visibility conditions. By employing a sonar-specific encoder and a hierarchical fusion approach that adapts to the quality of the sensing environment, SonarLLM significantly improves performance in tasks such as recognition, counting, and visual question answering. The model achieves a macro accuracy of 72.0% in sonar-only tasks and 68.7% in fusion tasks, demonstrating a substantial advantage over existing baselines, particularly in turbid conditions where sonar's complementary value increases.
SonarLLM outperforms traditional optical-centric models by leveraging sonar's unique strengths, achieving up to 36% better performance in challenging underwater environments.
Reliable underwater perception requires complementary sensing under variable visibility. Optical cameras capture appearance and semantics but degrade rapidly with turbidity, whereas imaging sonar preserves geometry while exhibiting distinct range-azimuth structure and acoustic artifacts. Existing MLLMs, built primarily on optical encoders, are therefore ill-suited to model sonar or adaptively exploit sonar-optical complementarity. We propose SonarLLM, a sonar-optical MLLM that treats sonar as a native perceptual modality. It combines a sonar-specific encoder, modality-specific physics-aware feature enhancement, and reliability-aware hierarchical fusion to align acoustic structure with optical semantics and dynamically adjust their contributions as sensing quality changes. We also introduce SonarBench, a paired benchmark that spans four tasks: recognition, counting, visual question answering, and captioning; and, across the benchmark, three input settings: sonar-only, optical-only, and fusion. By fixing the scene and sonar observation while varying optical degradation, SonarBench enables controlled measurement of cross-modal complementarity. SonarLLM achieves 72.0% macro accuracy across sonar-only recognition, counting, and VQA, outperforming the strongest baseline by 34.4 percentage points, and 68.7% under fusion, exceeding the best baseline by 25.1 points. For recognition and counting, the fusion-over-optical gain grows from 6.0 to 36.0 points as turbidity increases, indicating the increasing complementary value of sonar under controlled optical degradation. Together, these results show that robust heterogeneous perception depends not only on adding sonar, but on representing and weighting it according to its sensing characteristics.