Search papers, labs, and topics across Lattice.
The paper introduces OmniSonic, a flow-matching-based diffusion framework for universal holistic audio generation from video and text, capable of synthesizing both on-screen and off-screen sounds, including speech. OmniSonic employs a TriAttn-DiT architecture with a Mixture-of-Experts (MoE) gating mechanism to effectively process and balance the contributions of on-screen environmental sound, off-screen environmental sound, and speech conditions. Evaluated on a newly constructed UniHAGen-Bench benchmark, OmniSonic surpasses state-of-the-art methods in generating comprehensive auditory scenes.
Finally, a single model can generate realistic and comprehensive auditory scenes from video and text, including both environmental sounds and human speech, outperforming previous specialized approaches.
In this paper, we propose Universal Holistic Audio Generation (UniHAGen), a task for synthesizing comprehensive auditory scenes that include both on-screen and off-screen sounds across diverse domains (e.g., ambient events, musical instruments, and human speech). Prior video-conditioned audio generation models typically focus on producing on-screen environmental sounds that correspond to visible sounding events, neglecting off-screen auditory events. While recent holistic joint text-video-to-audio generation models aim to produce auditory scenes with both on- and off-screen sound but they are limited to non-speech sounds, lacking the ability to generate or integrate human speech. To overcome these limitations, we introduce OmniSonic, a flow-matching-based diffusion framework jointly conditioned on video and text. It features a TriAttn-DiT architecture that performs three cross-attention operations to process on-screen environmental sound, off-screen environmental sound, and speech conditions simultaneously, with a Mixture-of-Experts (MoE) gating mechanism that adaptively balances their contributions during generation. Furthermore, we construct UniHAGen-Bench, a new benchmark with over one thousand samples covering three representative on/off-screen speech-environment scenarios. Extensive experiments show that OmniSonic consistently outperforms state-of-the-art approaches on both objective metrics and human evaluations, establishing a strong baseline for universal and holistic audio generation. Project page: https://weiguopian.github.io/OmniSonic_webpage/
