Search papers, labs, and topics across Lattice.
This paper introduces DGCM-AVS, a tri-modal framework that enhances audio-visual segmentation by integrating depth information as a spatial structural cue. By employing a Depth-Aware Dynamic Modulator and Depth-Guided Progressive Fusion, the model effectively improves the separation of adjacent objects and aligns audio with visual features. The results demonstrate significant performance gains over existing methods, with improvements of 10.2% in M_J and 8.7% in M_F on the AVSS dataset, underscoring the importance of depth in multimodal perception tasks.
Depth integration in audio-visual segmentation leads to over 10% performance gains, revealing a critical yet overlooked modality in multimodal perception.
Audio-Visual Segmentation (AVS) is a fundamental task in multimodal perception that performs pixel-level segmentation of sounding objects in videos by leveraging both visual and audio cues. It has broad applications in video understanding, human-computer interaction, and autonomous driving. However, most existing AVS methods do not explicitly model geometric cues such as relative distance and occlusion, thereby limiting the robustness of cross-modal alignment. In human perception, spatial structure is naturally integrated with audio-visual evidence to accurately localize sounding objects. Motivated by this, we incorporate estimated depth as a spatial structural cue for AVS and propose DGCM-AVS, a tri-modal framework that jointly models audio, visual, and depth information. Specifically, we design a Depth-Aware Dynamic Modulator to improve the separation of adjacent objects while preserving intra-object feature consistency. Furthermore, we propose Depth-Guided Progressive Fusion, which uses depth as an intermediate bridge to progressively align audio cues with visual features. Compared to state-of-the-art methods, DGCM-AVS achieves relative improvements of 10.2 percent in M_J and 8.7 percent in M_F on the AVSS dataset. We believe our study highlights depth as a promising yet underexplored modality for AVS and may encourage further research in this direction.
