Search papers, labs, and topics across Lattice.
This paper introduces Teacher-Guided Dual-Path (TG-DP), a novel audio-visual representation learning framework that decouples masked reconstruction and contrastive alignment into separate optimization paths to mitigate semantic noise. TG-DP uses a teacher model to guide the organization of visible tokens in the contrastive branch, improving cross-modal alignment. Experiments on AudioSet demonstrate state-of-the-art zero-shot retrieval performance, with R@1 improving from 35.2% to 37.4% for video-to-audio retrieval and from 27.9% to 37.1% for audio-to-video retrieval.
Decoupling masked reconstruction and contrastive alignment in audio-visual representation learning yields surprisingly large gains in zero-shot retrieval, outperforming SOTA by a significant margin.
Recent advances in audio-visual representation learning have shown the value of combining contrastive alignment with masked reconstruction. However, jointly optimizing these objectives in a single forward pass forces the contrastive branch to rely on randomly visible patches designed for reconstruction rather than cross-modal alignment, introducing semantic noise and optimization interference. We propose TG-DP, a Teacher-Guided Dual-Path framework that decouples reconstruction and alignment into separate optimization paths. By disentangling the masking regimes of the two branches, TG-DP enables the contrastive pathway to use a visibility pattern better suited to cross-modal alignment. A teacher model further provides auxiliary guidance for organizing visible tokens in this branch, helping reduce interference and stabilize cross-modal representation learning. TG-DP achieves state-of-the-art performance in zero-shot retrieval. On AudioSet, it improves R@1 from 35.2\% to 37.4\% for video-to-audio retrieval and from 27.9\% to 37.1\% for audio-to-video retrieval. The learned representations also remain semantically robust, achieving state-of-the-art linear-probe performance on AS20K and VGGSound. Taken together, our results suggest that decoupling multimodal objectives and introducing teacher-guided structure into the contrastive pathway provide an effective framework for improving large-scale audio-visual pretraining. Code is available at https://github.com/wanglg20/TG-DP.
