Search papers, labs, and topics across Lattice.
This paper introduces MJEPA, a novel joint-embedding predictive architecture designed for audio-visual learning that utilizes a single unified encoder for both modalities. By employing a single predictive objective that operates across modalities, MJEPA enhances cross-modal synergy, which is crucial for effective representation learning. The results demonstrate that MJEPA significantly outperforms existing methods, achieving over 6.8 mAP improvement on AudioSet-20K and surpassing fully finetuned models on multiple benchmarks while using substantially less data.
Cross-modal prediction in MJEPA not only boosts performance but also reveals that a unified encoder can outperform traditional modality-specific approaches by leveraging shared information.
Self-supervised learning from large-scale video data has emerged as a dominant paradigm for visual representation learning. Since audio and visual streams naturally co-occur in video data, extending this success to jointly learn from both modalities is a natural next step, yet it remains challenging. Existing audio-visual self-supervised methods rely on modality-specific encoders and complex combinations of contrastive or reconstruction objectives, limiting cross-modal synergy and scalability. Joint Embedding Predictive Architectures (JEPAs) offer a simple, modality-agnostic alternative, but have to date been applied primarily to individual modalities. We introduce MJEPA, a joint-embedding predictive architecture for audio-visual learning that uses a single, unified encoder for both modalities. Our approach uses only a single predictive objective, applied both within and across modalities. We show that cross-modal prediction is critical: without it, a shared encoder degrades below unimodal baselines; with it, each modality's representation benefits from the other. Our frozen ViT-g model outperforms the best prior frozen baseline by over 6.8 mAP on AudioSet-20K, surpasses fully finetuned models on ESC-50 and FSD50K, and is competitive on video benchmarks despite using 10x less video data.
Meta AI