Search papers, labs, and topics across Lattice.
This paper introduces MiDashengLM-Gen, an innovative end-to-end framework that integrates a pre-trained Large Language Model (LLM) with per-token conditional flow matching for generating coherent audio scenes that combine speech, music, and sound effects. By addressing the limitations of traditional disjointed pipelines, the model significantly enhances speech intelligibility, achieving a Word Error Rate (WER) of 2.79% on the Seed-TTS benchmark, which is a substantial improvement over existing unified models. Additionally, MiDashengLM-Gen demonstrates strong performance in multilingual settings and maintains high-quality mixed-audio generation on the MECAT benchmark.
Achieving a staggering reduction in Word Error Rate from 12.15% to 2.79%, MiDashengLM-Gen sets a new standard for text-to-audio generation.
Generating coherent audio scenes that simultaneously blend speech, music, and sound effects remains a significant challenge. Current approaches typically rely on a disjointed pipeline where a frozen, decoupled text encoder feeds a separate audio decoder, limiting cross-modal optimization and leading to poor speech intelligibility. To overcome these limitations, we introduce MiDashengLM-Gen, an end-to-end framework that couples a pre-trained Large Language Model (LLM) with per-token conditional flow matching for autoregressive, variable-length mixed-audio scene generation. MiDashengLM-Gen represents a first approach for general text-to-audio generation with one end-to-end trained model. Empirical evaluations demonstrate that MiDashengLM-Gen drastically improves speech intelligibility over existing unified models. On the Seed-TTS benchmark, English Word Error Rate (WER) drops from 12.15% to 2.79%, approaching the performance of dedicated Text-to-Speech (TTS) systems (1.24%). Furthermore, the framework extends effectively to multilingual settings, yielding highly competitive multilingual WERs compared to existing baselines. Lastly, the model maintains competitive mixed-audio generation quality on the MECAT benchmark. Code and checkpoints are available at https://github.com/xiaomi-research/midashenglm-gen and https://huggingface.co/mispeech/midashenglm-gen, and the demo page is available at https://xingws.github.io/midashenglm-gen-demo/.
