Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces a training-free sound source localization (SSL) framework, GAR, that leverages the meta-reasoning capabilities of Multimodal Large Language Models (MLLMs). GAR employs a three-stage pipeline: Generation of bounding box proposals and audio classifications, Analysis of audio-visual consistency using role tagging and anchor voting, and Refinement via adaptive gating. Experiments on single and multi-source benchmarks demonstrate that GAR achieves competitive performance without requiring any training.
Skip the training data: MLLMs can directly reason about audio-visual relationships to localize sound sources, achieving competitive performance in complex acoustic scenes.
Sound source localization task aims to identify the locations of sound-emitting objects by leveraging correlations between audio and visual modalities. Most existing SSL methods rely on contrastive learning-based feature matching, but lack explicit reasoning and verification, limiting their effectiveness in complex acoustic scenes. Inspired by human meta-cognitive processes, we propose a training-free SSL framework that exploits the intrinsic reasoning capabilities of Multimodal Large Language Models (MLLMs). Our Generation-Analysis-Refinement (GAR) pipeline consists of three stages: Generation produces initial bounding boxes and audio classifications; Analysis quantifies Audio-Visual Consistency via open-set role tagging and anchor voting; and Refinement applies adaptive gating to prevent unnecessary adjustments. Extensive experiments on single-source and multi-source benchmarks demonstrate competitive performance. The source code is available at https://github.com/VisualAIKHU/GAR-SSL.