Search papers, labs, and topics across Lattice.
WorldSonus addresses the critical gap of audio synthesis in visual world models by introducing a real-time, interactive, and spatially aligned video-to-audio framework. It employs a streaming causal autoregressive diffusion model for low-latency generation (RTF 0.41) and an audio-centric captioning pipeline with chunk-indexed prompt scheduling for interactive control. Experiments show WorldSonus not only excels in world model applications but also generalizes to open-domain benchmarks, achieving state-of-the-art performance in acoustic quality and spatial alignment.
Interactive AI worlds can now speak and react in real-time with spatially accurate sound, bridging the sensory gap in generative environments.
Recent advances in world models have enabled increasingly realistic visual synthesis. However, these generated environments remain largely silent. Bringing sound to world models poses three core challenges: real-time generation to keep pace with interactive video streams, interactive control to respond to mid-stream sound instructions, and spatially aligned stereo to reflect scene geometry and camera motion. To address these demands, we introduce WorldSonus, an interactive video-to-audio framework designed for real-time spatial sound synthesis in world models. For real-time generation, WorldSonus employs a streaming causal autoregressive diffusion architecture that synthesizes audio chunks at a low real-time factor (RTF) of 0.41. For interactive control, we incorporate an audio-centric captioning pipeline with chunk-indexed prompt scheduling, enabling dynamic manipulation of sound events during generation. For spatial alignment, we leverage high-quality stereo supervision curated from diverse stereo and ambisonic data. Extensive experiments demonstrate that while tailored for world models, WorldSonus generalizes effectively to open-domain video-to-audio benchmarks, matching or outperforming state-of-the-art bidirectional models in both acoustic quality and spatial alignment. Project page: https://noizai.github.io/WorldSonus/
