Search papers, labs, and topics across Lattice.
This paper introduces SpeechAgent-R, a multimodal audio agent designed to tackle complex acoustic problems by coordinating intrinsic understanding with external tools and skills. The authors constructed the HIU-Corpus, a comprehensive dataset of 65,492 interaction trajectories and 507.6 hours of audio, to facilitate the training and evaluation of the agent's performance across various tasks. SpeechAgent-R significantly outperforms its base model, achieving an improvement of 15.40 points on in-distribution tasks and 14.23 points on out-of-distribution tasks, highlighting the effectiveness of skill and tool coordination in audio reasoning.
SpeechAgent-R's ability to adaptively coordinate skills and tools leads to a remarkable 15-point performance boost in complex audio reasoning tasks.
Complex acoustic problems may require models to perform acoustic operations, interact with external tools and reason over the resulting textual or processed-audio observations rather than answer directly from a fixed audio input. We study such problems as tool-interactive audio reasoning and develop SpeechAgent-R, an audio agent that coordinates its intrinsic multimodal understanding with external skills and tools. To support this capability, we construct HIU-Corpus, comprising 65,492 interaction trajectories and 507.6 hours of audio across 24 tasks, 8 skills and 9 tools. SpeechAgent-R first learns structured interaction behaviors through trajectory-based supervised fine-tuning and then improves its decisions through multi-turn reinforcement learning. We further introduce HIU-Bench to jointly evaluate task performance, interaction quality and generalization to diverse task settings. It contains 1,395 samples across 56 tasks, including in-distribution (ID) and out-of-distribution (OOD) splits with substantial shifts in tool usage and workflow composition. SpeechAgent-R achieves 84.17 on ID tasks and 70.94 on OOD tasks, improving over the base model under the same agent harness by 15.40 and 14.23 points. These results demonstrate that learning skill and tool coordination improves audio agents'ability to handle diverse task settings and adaptive tool interactions.
