Search papers, labs, and topics across Lattice.
This paper investigates the extent to which LLMs, pre-trained on text alone, possess auditory knowledge and how this impacts their performance when adapted into Large Audio Language Models (LALMs). They evaluate LLMs using direct probing with the AKB-2000 auditory knowledge benchmark, cascade evaluation using audio captioners, and fine-tuning into LALMs. The study reveals significant variance in auditory knowledge across different LLM families and a strong correlation between text-only auditory knowledge and downstream audio task performance.
LLMs' text-only pre-training secretly encodes surprisingly different levels of auditory knowledge, directly impacting their effectiveness as backbones for audio language models.
Large language models (LLMs) have been widely used as knowledge backbones of Large Audio Language Models (LALMs), yet how much auditory knowledge they encode through text-only pre-training and how this affects downstream performance remains unclear. We study this gap by comparing different LLMs under two text-only and one audio-grounded setting: (1) direct probing on AKB-2000, a curated benchmark testing the breadth and depth of auditory knowledge; (2) cascade evaluation, where LLMs reason over text descriptions from an audio captioner; and (3) audio-grounded evaluation, where each LLM is fine-tuned into a Large Audio Language Model (LALM) with an audio encoder. Our findings reveal that auditory knowledge varies substantially across families, and text-only results are strongly correlated with audio performance. Our work provides empirical grounding for a comprehensive understanding of LLMs in audio research.
