Search papers, labs, and topics across Lattice.
This paper addresses the challenge of localizing multiple sound sources in visual scenes by leveraging a self-supervised dual-source audio-visual learning framework that utilizes selective convergence. The authors demonstrate that contrastive models can naturally focus on the most salient audio-visual correspondences, effectively breaking the circular dependency between audio separation and source localization. Their approach not only outperforms existing self-supervised methods on dual-source benchmarks but also introduces a novel evaluation method using pixel-level segmentation masks to mitigate biases in current benchmarks.
Selective convergence in self-supervised learning allows models to prioritize salient audio-visual cues, leading to superior performance in multi-source localization without manual annotations.
Localising multiple sound sources in visual scenes remains a fundamental challenge in multimodal perception due to an inherent circular dependency: separating mixed audio requires knowing source locations, while identifying sound-producing regions requires separated audio signals. In this paper, we focus on the dual-source setting and discover a selective convergence in self-supervised audio-visual learning: when presented with multiple sound sources, contrastive models naturally converge to the most salient audio-visual correspondence rather than attempting to represent all sources equally. This emergent phenomenon, analogous to human selective auditory attention, enables us to break the above circular dependency through a progressive two-stage framework: first, leveraging selective convergence to identify dominant sources, and then exploiting these learned priors to uncover remaining sources. Our self-supervised approach achieves the best performance among self-supervised methods on dual-source benchmarks without requiring any manual annotations, and even surpasses some weakly-supervised approaches \red{on certain metrics. Furthermore, we identify a fundamental evaluation inconsistency in existing benchmarks: comparing continuous localisation heatmaps against bounding-box annotations creates systematic biases, particularly for non-axis-aligned objects where the bounding box includes substantial background regions. To address this, we introduce pixel-level segmentation masks to the existing benchmark, enabling spatially-aligned evaluation. Together, these results suggest that embracing rather than suppressing selectivity offers a scalable, annotation-free route to multi-source localisation.
