Search papers, labs, and topics across Lattice.
The paper introduces audio-interleaved reasoning for Large Audio Language Models (LALMs) to overcome the information bottleneck of one-time audio encoding. They propose a two-stage training framework involving supervised fine-tuning for salient audio segment localization and reinforcement learning to encourage re-listening. The resulting LALM, Echo, demonstrates improved performance on audio comprehension benchmarks, showcasing the benefits of dynamic audio re-listening during reasoning.
LALMs can now dynamically re-listen to audio during reasoning, leading to state-of-the-art performance on complex audio comprehension tasks.
The maturation of Large Audio Language Models (LALMs) has raised growing expectations for them to comprehend complex audio much like humans. Current efforts primarily replicate text-based reasoning by contextualizing audio content through a one-time encoding, which introduces a critical information bottleneck. Drawing inspiration from human cognition, we propose audio-interleaved reasoning to break through this bottleneck. It treats audio as an active reasoning component, enabling sustained audio engagement and perception-grounded analysis. To instantiate it, we introduce a two-stage training framework, first teaching LALMs to localize salient audio segments through supervised fine-tuning, and then incentivizing proficient re-listening via reinforcement learning. In parallel, a structured data generation pipeline is developed to produce high-quality training data. Consequently, we present Echo, a LALM capable of dynamically re-listening to audio in demand during reasoning. On audio comprehension benchmarks, Echo achieves overall superiority in both challenging expert-level and general-purpose tasks. Comprehensive analysis further confirms the efficiency and generalizability of audio-interleaved reasoning, establishing it as a promising direction for advancing audio comprehension. Project page: https://github.com/wdqqdw/Echo.
Tsinghua AI