Search papers, labs, and topics across Lattice.
Foley-Omni is a unified multimodal audio generation model that advances beyond isolated task-level synthesis to generate complete video soundtracks by jointly modeling speech, sound effects, and music within a shared latent generation process. This approach addresses the need for coherent audio tracks in video production, enabling more consistent and high-quality audio outputs. Experimental results demonstrate that Foley-Omni not only competes with expert systems in individual tasks but also enhances speech intelligibility and overall perceptual quality in mixed soundtrack generation.
Foley-Omni achieves expert-level performance in audio synthesis while generating cohesive soundtracks for video, enhancing both intelligibility and quality.
Recent unified audio generation models can support diverse tasks across speech, sound effects, and music, but most of them still focus on isolated task-level synthesis. However, real video production often requires multiple components of a complete audio track to be generated jointly and consistently for the same video. We present Foley-Omni, a unified multimodal audio generation model that extends isolated task-level synthesis to complete video soundtrack generation by jointly modeling speech, sound effects, and music within a shared latent generation process. To support training and reproducible evaluation, we develop an audiovisual data curation pipeline and introduce V2ST-Bench, a benchmark for holistic video soundtrack generation evaluation. Experiments show that Foley-Omni achieves competitive performance with expert systems on individual synthesis tasks, while improving speech intelligibility, audiovisual consistency and perceptual quality for mixed soundtrack generation.
