Search papers, labs, and topics across Lattice.
This paper introduces ScenA, a novel approach to multi-speaker audio scene generation that leverages a text-to-audio flow-matching foundation model trained on diverse in-the-wild data. By conditioning the model on reference voices and free-form natural language prompts, ScenA generates rich audio scenes that incorporate ambient sounds and overlapping dialogue without the need for structured supervision. The method addresses the challenge of the "Reference Shortcut" by employing a high-noise-biased timestep distribution, resulting in superior performance on speaker-binding metrics compared to existing systems.
ScenA reveals that conditioning on free-form prompts and reference voices can produce more authentic multi-speaker audio scenes than traditional structured methods.
Existing multi-speaker dialogue systems bind speakers to utterances through structured supervision: per-turn tags, multi-stream transcriptions, or learnable speaker embeddings. These systems operate within speech-only pipelines that produce clean vocal sequences without the ambient texture of real conversations. We take a different approach. Our method, ScenA, conditions a text-to-audio flow-matching foundation model, pretrained on large-scale in-the-wild data, directly on multiple reference voices and a free-form natural language prompt that describes an entire multi-speaker audio scene. Leveraging such a foundational model allows us to inherit its capacity for natural, non-studio audio: background noise, room acoustics, overlapping dialogue, and spontaneous paralinguistic events, while adding multi-speaker control without any per-turn structure. Concretely, reference latents are concatenated into the model's token sequence and distinguished by lightweight identity-aware positional encodings. However, we identify a critical obstacle to this approach: the \textit{Reference Shortcut}. During training under standard noise schedules, the model can identify the matching reference by acoustic similarity to the noisy target, bypassing the text prompt entirely. We address this with a high-noise-biased timestep distribution that forces the model to rely on the text prompt for speaker assignment. We evaluate ScenA on the CoVoMix2-Dialogue benchmark, showing that it outperforms existing multi-speaker systems on speaker-binding metrics while generating rich conversational audio with overlapping speech, emotional vocalizations, and ambient sound. Our results demonstrate the advantage of using a general-purpose audio model conditioned on a free-form scene description, rather than passing structured dialog scripts through a speech-only pipeline.
