Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The paper introduces HSC-MAE, a dual-path teacher-student masked autoencoder framework for unsupervised audio-visual representation learning from weakly paired data. It enforces semantic consistency across global, local, and sample levels using Deep Canonical Correlation Analysis (DCCA), teacher-mined soft top-k affinities, and masked autoencoding, respectively. Experiments on AVE and VEGAS datasets demonstrate significant improvements in mean Average Precision (mAP) compared to existing unsupervised methods, indicating the learned representations are more robust and well-structured.
Unsupervised audio-visual representation learning gets a boost from HSC-MAE, which cleverly aligns embeddings across multiple semantic levels, outperforming existing methods on weakly paired data.
Learning aligned multimodal embeddings from weakly paired, label-free corpora is challenging: pipelines often provide only pre-extracted features, clips contain multiple events, and spurious co-occurrences. We propose HSC-MAE (Hierarchical Semantic Correlation-Aware Masked Autoencoder), a dual-path teacher-student framework that enforces semantic consistency across three complementary levels of representation - from coarse to fine: (i) global-level canonical-geometry correlation via DCCA, which aligns audio and visual embeddings within a shared modality-invariant subspace; (ii) local-level neighborhood-semantics correlation via teacher-mined soft top-k affinities, which preserves multi-positive relational structure among semantically similar instances; and (iii) sample-level conditional-sufficiency correlation via masked autoencoding, which ensures individual embeddings retain discriminative semantic content under partial observation. Concretely, a student MAE path is trained with masked feature reconstruction and affinity-weighted soft top-k InfoNCE; an EMA teacher operating on unmasked inputs via the CCA path supplies stable canonical geometry and soft positives. Learnable multi-task weights reconcile competing objectives, and an optional distillation loss transfers teacher geometry into the student. Experiments on AVE and VEGAS demonstrate substantial mAP improvements over strong unsupervised baselines, validating that HSC-MAE yields robust and well-structured audio-visual representations.