Search papers, labs, and topics across Lattice.
This paper introduces a Unified Audio Front-end LLM (UAF) that integrates various audio front-end tasks, including VAD, TD, speaker recognition, ASR, and QA, into a single autoregressive sequence prediction framework. UAF processes streaming audio chunks with a reference audio prompt to anchor the target speaker, enabling the model to generate tokens encoding both semantic content and system-level state controls. Experiments show that UAF achieves state-of-the-art performance across multiple audio front-end tasks and improves response latency and interruption accuracy in full-duplex speech interaction.
Ditch the clunky pipeline: a single LLM can now handle all your audio front-end needs, slashing latency and boosting accuracy in full-duplex speech interactions.
Full-duplex speech interaction, as the most natural and intuitive mode of human communication, is driving artificial intelligence toward more human-like conversational systems. Traditional cascaded speech processing pipelines suffer from critical limitations, including accumulated latency, information loss, and error propagation across modules. To address these issues, recent efforts focus on the end-to-end audio large language models (LLMs) like GPT-4o, which primarily unify speech understanding and generation task. However, most of these models are inherently half-duplex, and rely on a suite of separate, task-specific front-end components, such as voice activity detection (VAD) and turn-taking detection (TD). In our development of speech assistant, we observed that optimizing the speech front-end is equally crucial as advancing the back-end unified model for achieving seamless, responsive interactions. To bridge this gap, we propose the first unified audio front-end LLM (UAF) tailored for full-duplex speech systems. Our model reformulates diverse audio front-end tasks into a single auto-regressive sequence prediction problem, including VAD, TD, speaker recognition (SR), automatic speech recognition (ASR) and question answer (QA). It takes streaming fixed-duration audio chunk (e.g., 600 ms) as input, leverages a reference audio prompt to anchor the target speaker at the beginning, and regressively generates discrete tokens encoding both semantic content and system-level state controls (e.g., interruption signals). Experiments demonstrate that our model achieves leading performance across multiple audio front-end tasks and significantly enhances response latency and interruption accuracy in real-world interaction scenarios.
