Search papers, labs, and topics across Lattice.
This paper introduces a novel training strategy, self-rephrasing, to adapt reasoning LLMs (RLMs) for audio-language tasks by converting self-generated responses into audio-understanding variants. The approach maintains distributional alignment and avoids unnatural responses caused by exposing textual surrogate inputs to the RLM's chain-of-thought reasoning. The authors also fuse and compress multiple audio encoders to create stronger audio representations and construct a large multi-task corpus for training, resulting in a 4B-parameter ALM that outperforms similarly sized models and surpasses larger ALMs on audio-reasoning benchmarks.
By cleverly "self-rephrasing" LLM outputs, this work coaxes reasoning LLMs to handle audio inputs without sacrificing their chain-of-thought abilities.
Large audio language models (ALMs) extend LLMs with auditory understanding. A common approach freezes the LLM and trains only an adapter on self-generated targets. However, this fails for reasoning LLMs (RLMs) whose built-in chain-of-thought traces expose the textual surrogate input, yielding unnatural responses. We propose self-rephrasing, converting self-generated responses into audio-understanding variants compatible with RLMs while preserving distributional alignment. We further fuse and compress multiple audio encoders for stronger representations. For training, we construct a 6M-instance multi-task corpus (2.5M unique prompts) spanning 19K hours of speech, music, and sound. Our 4B-parameter ALM outperforms similarly sized models and surpasses most larger ALMs on related audio-reasoning benchmarks, while preserving textual capabilities with a low training cost. Notably, we achieve the best open-source result on the MMAU-speech and MMSU benchmarks and rank third among all the models.
