Search papers, labs, and topics across Lattice.
This paper introduces Hear to See (H2S), a novel framework for audio-visual instance segmentation that effectively addresses the challenges of matching overlapping acoustic events with visual instances and tracking them amidst asynchronous audio-visual dynamics. H2S employs two key mechanisms: the Acoustic-Semantic Projector (ASP) for disentangling mixed audio and establishing hierarchical correspondences, and the Asynchronous Dynamics Modulator (ADM) for adaptive state transitions based on audio modulation. Experimental results on the AVISeg dataset demonstrate that H2S achieves state-of-the-art performance with a mean Average Precision (mAP) of 48.54, outperforming the previous best by 7.8%.
H2S achieves a remarkable 48.54 mAP in audio-visual instance segmentation, setting a new benchmark in the field.
Audio-visual instance segmentation (AVIS) requires accurately identifying and tracking individual sounding objects with pixel-level masks. Existing methods struggle to match overlapping acoustic events with visual instances and handle asynchronous audio-visual dynamics. Therefore, two critical questions arise: how can a model establish precise correspondence between overlapping sound sources and visual instances, and how can a model maintain robust tracking when audio and visual signals are temporally misaligned?This paper proposes Hear to See (H2S), addressing these challenges through two mechanisms. The Acoustic-Semantic Projector (ASP) disentangles mixed audio and establishes hierarchical correspondence from semantic to spatial domains. The Asynchronous Dynamics Modulator (ADM) adaptively adjusts state transitions via audio-modulated Mamba, prioritizing current information during dynamic variations and maintaining continuity in stable periods.Experiments on AVISeg show H2S achieves SOTA performance, attaining 48.54 mAP with a COCO pretrained ResNet50 and surpassing the previous by 7.8\%. The code will be open-sourced once the paper is accepted. The source code will be publicly available at https://github.com/leiyeliu/H2S.
CMU ML