Search papers, labs, and topics across Lattice.
This paper introduces SALMONN-2, an audio large language model (ALLM) that leverages self-supervised learning (SSL) audio representations as a foundation, challenging the reliance on supervised data for audio encoders. By employing a multi-layer feature fusion (MLF) adapter, SALMONN-2 effectively integrates hierarchical information from SSL encoders, enhancing performance across diverse audio tasks. The model not only matches or surpasses specialized supervised encoders but also achieves state-of-the-art results on ALLM understanding benchmarks, demonstrating the potential of SSL in advancing general-purpose hearing abilities.
General-purpose self-supervised audio representations can outperform specialized supervised models, reshaping the landscape of audio understanding in ALLMs.
Recent audio large language models (ALLMs) are typically built upon audio encoders trained with large amounts of supervised data. Since self-supervised learning (SSL) audio encoder models are known to learn general-purpose and transferable representations, we investigate whether general-purpose SSL audio representations can serve as an effective foundation for ALLMs. We present SALMONN-2, an ALLM built upon a unified SSL encoder. To better exploit the hierarchical representations learned by SSL encoders, we propose a multi-layer feature fusion (MLF) adapter that aggregates information from all encoder layers before projecting them into the language model. Beyond conventional audio understanding tasks, we further explore multimodal in-context learning (MICL) in ALLMs and study how this capability can be acquired through contextual biasing training. Experimental results show that a general-purpose SSL encoder achieves performance comparable to, or better than, specialised supervised audio encoders while providing a more balanced capability across speech, audio, music and paralinguistic tasks. SALMONN-2 further achieves state-of-the-art performance among comparable-scale open-weight models on ALLM understanding benchmarks, obtaining the best results on MMAU-Pro, MMAR and MMSU. We also show that MICL does not emerge naturally in ALLMs, but can be effectively acquired through targeted contextual biasing training.
