Search papers, labs, and topics across Lattice.
FireRedAudio is a novel general-purpose audio language model that utilizes a 9B-parameter LLM to simultaneously handle audio understanding and speech generation through decoupled continuous representations. By employing a dedicated Audio Encoder for recognition and a RedAE-based pathway for speech generation, the model achieves advanced capabilities in automatic speech recognition (ASR) and text-to-speech (TTS) tasks, including zero-shot and instructed TTS. Comprehensive evaluations reveal that FireRedAudio not only excels in multilingual ASR and audio understanding but also significantly outperforms existing models in semantic and acoustic speech editing, demonstrating the effectiveness of its unique representation strategy.
FireRedAudio achieves leading performance in audio understanding and speech generation by decoupling input representations, marking a significant advancement in unified audio-language modeling.
A unified audio model must recognize and understand linguistic, paralinguistic, and environmental information while supporting speech synthesis and editing. A key challenge is representation: understanding favors compact features suited to long-context modeling, whereas speech generation requires reconstructible features that preserve fine-grained acoustic detail. We introduce FireRedAudio, a general-purpose audio language model with a shared 9B-parameter LLM. To the best of our knowledge, it is the first publicly disclosed unified audio-language model to provide separate continuous input representations for understanding and generation within a single trainable autoregressive LLM. Audio to be recognized or analyzed is processed by a dedicated Audio Encoder, while speech inputs for generation use a RedAE-based pathway. The LLM directly generates text or conditions a flow-matching DiT to produce continuous acoustic latents. Through progressive multitask training, FireRedAudio supports ASR and audio understanding, with the latter extending to recordings of up to one hour, as well as zero-shot TTS, Instruct TTS, and semantic and acoustic speech editing. Its structured organization of long-form audio achieves second-level timestamp accuracy. Across comprehensive evaluations, FireRedAudio achieves competitive or leading performance in audio understanding and multilingual ASR, strong content accuracy and speaker preservation in zero-shot TTS, leading instruction following in Instruct TTS, and substantial improvements over Ming-UniAudio-Edit in both semantic and acoustic speech editing. These results demonstrate the viability of decoupled continuous input representations for unifying audio understanding and continuous-latent speech generation in a model of moderate scale. Our code is available at https://github.com/FireRedTeam/FireRedAudio.
