Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The paper introduces VoxParadox, an adversarial benchmark designed to evaluate Audio LLMs' understanding of paralinguistic information by creating controlled speech synthesis examples where transcript claims and speaking style are intentionally mismatched. Evaluations reveal that Audio LLMs struggle with acoustic ground truth, often prioritizing language-implied answers, and that paralinguistic cues degrade in deeper encoder layers. To mitigate this, the authors propose Prompt-Conditioned Layer Mixer (PCLM) combined with Direct Preference Optimization (DPO), significantly improving paralinguistic understanding in Audio LLMs.
Audio LLMs are often fooled by what they "read" in the transcript, ignoring critical paralinguistic cues in the audio itself, but a simple layer mixing strategy can dramatically improve their acoustic awareness.
Audio large language models (Audio LLMs) demonstrate strong performance on speech understanding tasks, yet their ability to understand paralinguistic information remains limited. To systematically quantify this issue, we introduce VoxParadox, an adversarial benchmark with 2,000 verified examples, spanning 10 paralinguistic tasks, created with controlled speech synthesis to intentionally mismatch transcript claims and speaking style, enabling direct measurement of speech paralinguistic understanding. Evaluation of a diverse set of Audio LLMs reveals consistently low accuracy on acoustic ground truth and a strong tendency to follow language-implied (incorrect) answers. To understand the cause of this gap, we perform layer-wise probing and find that (i) paralinguistic cues can degrade in deeper encoder layers and at the encoder--LLM interface, and (ii) even when such cues are available in audio tokens, the language model frequently ignores them. To address these problems, we propose Prompt-Conditioned Layer Mixer (PCLM), which adaptively combines information from multiple audio layers based on the input prompt, and pair it with Direct Preference Optimization (DPO) to explicitly prefer acoustically supported options over language-implied alternatives. These methods substantially improve Audio LLM paralinguistic understanding, improving Audio Flamingo 3 from 17.40% to 65.20% on VoxParadox, and from 37.74% to 54.78% on MMSU paralinguistic subset. Our project page is available at https://voxparadox.github.io/.