Search papers, labs, and topics across Lattice.
AuK unifies instruction-guided speech generation and arbitrary audio editing into a single open-source foundational model trained on 1.95 million hours of audio and 3.03 billion instruction-audio instances across five core task families. The architecture couples an MLLM semantic prior with a hybrid dual- and single-stream rectified-flow Transformer, optimized through staged pre-training and distinct post-training regimes (RLHF for open-ended editing and reward-driven RL for synthesis). Distilled via consistency initialization and task-routed Decoupled DMD, the resulting AuK-Flash executes CFG-free generation in just 4 steps with a 4.5脳 wall-clock speedup while matching or outperforming state-of-the-art baselines.
Natural-language instructions can now handle both zero-shot speech synthesis and surgical acoustic editing within a single unified model, operating at 4-step distilled inference speeds without classifier-free guidance.
We introduce AuK, an open-source foundational model that unifies speech generation and editing through a common interface of natural-language instructions and audio context. To support this broad capability set, we construct approximately 3.03 billion instruction--audio instances and 1.95 million hours of effective supervision across five task families: speech generation, content editing, enhancement and separation, paralinguistic editing, and acoustic editing. AuK combines a multimodal large language model for semantic conditioning, an VAE jointly trained on speech, general audio, and music for acoustic conditioning, and a hybrid rectified-flow Transformer that performs dual-stream MMDiT blocks followed by unified single-stream DiT blocks for generation. Training begins with generation-only warm-up and proceeds to joint generation--editing pre-training. We then apply complementary post-training strategies: human-feedback preference optimization for open-ended editing and reward-based reinforcement learning for speech generation. To reduce inference cost, we further distill the model with consistency initialization and task-routed Decoupled DMD. The resulting AuK-Flash performs 4-step inference without classifier-free guidance and achieves a 4.5 wall-clock speedup over the full model under matched conditions. Experiments demonstrate leading performance on zero-shot and instruction-controlled speech generation and general instruction-guided editing, while remaining competitive on signal-level restoration tasks. We release both the source code and model weights to support reproducibility and further research.
Tsinghua AI