Search papers, labs, and topics across Lattice.
This paper introduces a novel framework for integrating trainable prompts into the audio encoder of Audio-Language Models (ALMs) to enhance few-shot learning capabilities. By focusing on task-specific acoustic feature extraction, the authors demonstrate that this approach, when combined with existing text-side prompt tuning, leads to significant performance improvements across 11 datasets. The results indicate that modulating the audio representation space can effectively complement traditional text-only prompting methods, thereby advancing the state of few-shot adaptation in ALMs.
Integrating trainable prompts into the audio encoder can significantly boost few-shot learning performance in Audio-Language Models, outperforming traditional text-only approaches.
Audio-Language Models (ALMs) have shown remarkable success in zero-shot audio classification by aligning audio waveforms with text. Recent efforts to improve downstream performance focus on learning optimal text prompts. However, previous approaches focus on the text encoder, leaving the potential of learnable prompts within the audio encoder unexplored. In this paper, we propose a novel framework that introduces trainable prompts into the audio encoder to capture task-specific acoustic features. We demonstrate that integrating audio-side prompt learning with existing text-side approaches enhances few-shot adaptation. Through extensive experiments across 11 datasets show that integrating our method as a plug-and-play module alongside existing text prompt tuning generally leads to performance improvements. These findings suggest that explicitly modulating the audio representation space effectively complements text-only prompting approaches. The code is available at https://github.com/hyebin-c/aspl.
