Search papers, labs, and topics across Lattice.
This paper introduces COMET, a PLS-SVD framework, to dissect the modality gap in CLAP models, revealing that only a small subset of concept axes significantly contributes to similarity computation. They find that the modality gap is not solely attributable to the cone effect (mean shift), as previously thought. Based on these insights, they propose a spectral truncation method that mitigates the modality gap in a training-free manner, improving zero-shot audio captioning and enabling dimensionality reduction.
CLAP models suffer from a modality gap that's more than just a "cone effect" – it's a small set of interpretable concept axes dominating similarity, and truncating the rest unlocks near-supervised zero-shot performance.
Contrastive Language-Audio Pretraining (CLAP) models are widely used for audio understanding and support modality-agnostic condition swapping in many zero-shot applications. However, their performance is heavily affected by the modality gap between audio and text embeddings. Existing explanations mainly attribute this gap to the cone effect, treating it as a shift between mean embeddings, yet correcting the mean alone yields only limited improvements. Alternative hypotheses, such as information imbalance and dimensionality collapse, have also been proposed, but they remain insufficiently verified and have not been thoroughly studied in the audio domain. Meanwhile, several works attempt to decompose multimodal contrastive embeddings into interpretable concepts, but none explicitly analyze the modality gap from the perspective of concept decomposition. In this work, we introduce COMET (Concept space Organization and Modality gap Explanation with PLS-SVD Transformation), a novel partial least squares singular value decomposition (PLS-SVD) framework for CLAP that unveils a broader perspective of the modality gap. Our framework reveals that only a small, interpretable subset of axes, which captures shared concepts, contributes substantially to similarity computation, and that the mean component represents only partially the modality gap. Building on this insight, we propose a simple spectral truncation method that mitigates the modality gap in a training-free manner. The method enables zero-shot audio captioning with condition swapping to approach fully supervised performance, without requiring large auxiliary memory banks or expensive computation. At the same time, it achieves substantial embedding dimensionality reduction while preserving strong performance on retrieval and audio captioning tasks.
