Search papers, labs, and topics across Lattice.
This paper introduces Text-Prompted CLAP (TP-CLAP), an enhancement of the Contrastive Language-Audio Pretraining (CLAP) model that integrates a cross-attention-based fusion module to better align text and audio representations. By employing an audio multiple-choice question framework, TP-CLAP effectively learns to condition audio representations on textual prompts, leading to improved performance in audio question answering and retrieval tasks. Experimental results reveal that TP-CLAP not only surpasses the original CLAP model but also competes favorably against larger audio language models, particularly in music retrieval scenarios.
TP-CLAP achieves superior audio understanding by seamlessly integrating text prompts, outperforming traditional models in complex retrieval tasks.
Contrastive Language-Audio Pretraining (CLAP) learns aligned text and audio representations in a shared embedding space. However, independent encoding of each modality limits its ability to model cross-modal semantics in complex audio understanding and retrieval tasks. To address this limitation, this paper proposes Text-Prompted CLAP (TP-CLAP), a parameter-efficient extension of CLAP that introduces a cross-attention-based fusion module to incorporate textual prompts into audio features. TP-CLAP is trained using an audio multiple-choice question (audio-MCQ) framework, where it learns to align query-conditioned audio representations with text embeddings of correct answer choices via contrastive learning. Experiments demonstrate that TP-CLAP achieves competitive performance on audio question answering (audio-QA) with substantially larger audio-LLMs, while also improving the base CLAP model on audio-text retrieval and zero-shot classification benchmarks. The learned representations are further fine-tuned for attribute-focused audio-to-audio retrieval, showing that TP-CLAP consistently outperforms the standard CLAP baseline in music retrieval tasks.
