Search papers, labs, and topics across Lattice.
This paper introduces Audex, a unified audio-text large language model (LLM) that leverages a single Transformer decoder to seamlessly integrate audio and text inputs, enhancing multimodal generation capabilities. By meticulously curating extensive audio-text datasets and employing a multi-stage training approach, Audex achieves state-of-the-art performance across various audio intelligence tasks, including speech recognition and audio generation, without compromising the reasoning and alignment strengths of its text-only predecessor. The model's architecture and training methodology ensure compatibility with existing LLM infrastructure, making it a significant advancement in audio intelligence research.
Audex achieves state-of-the-art audio understanding and generation while maintaining the reasoning prowess of its text-only foundation, all through a unified architecture.
Audio intelligence involves understanding, reasoning about, and generating both audio and speech. In this work, we introduce Nemotron-Labs-Audex-30B-A3B (Audex), a unified audio-text LLM built on Nemotron-Cascade-2-30B-A3B, a strong text-only MoE LLM. Audex adopts a simple unified design with a single Transformer decoder: audio inputs are encoded and projected into the text embedding space, while text tokens and quantized audio output tokens are treated uniformly during generation. This architecture enables strong audio-text fusion, seamless multimodal generation, and compatibility with standard LLM training and inference infrastructure. For training, we meticulously curate audio-text datasets comprising 157.4B audio tokens and 320.5B text tokens. We apply multi-stage supervised training on these datasets, followed by text-only Cascade RL and multi-domain on-policy distillation. Audex delivers state-of-the-art audio understanding, speech recognition and translation, text-to-speech, audio generation, and speech-to-speech generation, while preserving very compelling reasoning, alignment, knowledge, long-context, and agentic capabilities of its text-only LLM backbone with marginal or no regression. We release the model checkpoints to facilitate open research.
NVIDIA