Search papers, labs, and topics across Lattice.
This paper introduces a learnable wait-think-answer control formulation for Large Audio-Language Models (LALMs) to balance reasoning quality and responsiveness in real-time spoken interaction. The controller decides, based on partial audio evidence, when to wait, externalize a reasoning update, or answer. Trained with supervised fine-tuning (SFT) and Decoupled Clip and Dynamic Sampling Policy Optimization (DAPO) using a composite reward, the controller improves accuracy and reduces post-endpoint deliberation time on synthetic and real audio benchmarks using Qwen2.5-Omni-7B.
LALMs can learn to strategically "think aloud" during spoken interactions, improving accuracy and reducing response latency by externalizing intermediate reasoning steps.
Recent advances in Large Audio-Language Models (LALMs) have made real-time, streaming spoken interaction increasingly practical. In this setting, reasoning quality and responsiveness are tightly coupled: delaying reasoning until the speech endpoint can improve answer quality but moves deliberation into user-visible response delay, while answering too early risks committing before decisive evidence arrives. We introduce a learnable wait-think-answer control formulation for LALMs. Motivated by the incremental nature of human conversation, the controller decides under partial audio evidence when to wait, when to externalize a compact reasoning update, and when to answer. Using Qwen2.5-Omni-7B as the base model, we construct aligned wait-think-answer traces from spoken reasoning data, train the controller with supervised fine-tuning (SFT), and then apply Decoupled Clip and Dynamic Sampling Policy Optimization (DAPO). The reward combines answer correctness, action validity, update timing, latency synchronization, reasoning quality, and chain consistency, optimizing the complete wait-think-answer trajectory and not the final answer alone. On a six-task synthetic spoken reasoning question answering (SRQA) benchmark, the six-reward DAPO controller improves the row-weighted accuracy from 67.6% to 70.3% while reducing post-endpoint final-think length by 14% under the same Qwen deployment harness. On a 186-item human-recorded Real Audio Bench, a transfer check beyond text-to-speech (TTS)-rendered speech, the controller family remains functional: SFT achieves the strongest accuracy, while the six-reward DAPO controller is the only learned variant whose final-think length falls below the base. These results suggest that a streaming model should learn when to make intermediate reasoning explicit during the audio stream.
