Search papers, labs, and topics across Lattice.
This paper introduces ZEBRA, a novel framework that enhances zero-shot performance in Audio-Language Models (ALMs) by integrating zero-shot logits with prompt-learning logits and applying self-entropy regularization. The authors identify a significant base-to-novel generalization gap in existing prompt learning methods, where performance on novel classes often deteriorates despite improvements in base class accuracy. Experimental results demonstrate that ZEBRA effectively narrows this gap, achieving superior novel-class performance while preserving strong accuracy on base classes across various audio classification datasets.
ZEBRA bridges the critical base-to-novel generalization gap in audio-language models, boosting novel-class performance without sacrificing base accuracy.
Audio-Language Models (ALMs) achieve strong zero-shot performance by aligning audio with textual class descriptions. Although prompt learning improves accuracy on base classes through few-shot supervised adaptation, we observe a critical trade-off: it often degrades performance on novel classes, sometimes falling below zero-shot accuracy. This exposes a base-to-novel generalization gap in prompt learning for ALMs. To address this issue, we propose \textbf{ZEBRA} (Zero-shot Entropy-Regularized Prompt Learning for Base-to-Novel Generalization), a plug-and-play framework that fuses zero-shot logits with prompt-learning logits, and employs self-entropy regularization to reduce overfitting to base classes. Experiments across multiple audio classification datasets show that ZEBRA consistently improves novel-class performance while maintaining strong base accuracy, significantly reducing the base-to-novel gap compared to standard prompt learning. The code is available at: https://github.com/asif-hanif/zebra.
