Search papers, labs, and topics across Lattice.
This paper addresses the lack of dynamic spatial correspondence in current joint video-audio generation models, which is crucial for immersive applications. The authors introduce StereoBind, a framework that leverages visual motion tracks to enforce source-aware audiovisual correspondence, absolute positional encoding, and relative motion modeling for coherent stereo audio generation. Experiments on the new StereoWorld-29K dataset demonstrate StereoBind's significant improvement in spatial alignment of stereo audio without compromising overall audiovisual quality.
Immersive AR/VR experiences can finally have stereo audio that dynamically tracks visual sources, thanks to a novel framework that binds visual motion to sound localization.
Recent joint video-audio generation models have achieved strong semantic correspondence and temporal synchronization. However, applications such as AR/VR and interactive gaming further require stereo audio to provide an immersive sense, which remains largely overlooked. Effective stereo audio requires the perceived sound location to evolve consistently with the motion of its corresponding visual source. We refer to this property as Dynamic Spatial Correspondence and propose StereoBind, a framework that binds visual source motion to stereo sound generation. StereoBind uses motion tracks to coordinate visual motion and stereo audio through three complementary mechanisms. Visual Motion Binding establishes source-aware audiovisual correspondence, the Spatial Track Encoder captures absolute source positions, and Residual Track RoPE models relative motion. For supervision and evaluation, we construct StereoWorld-29K, a large-scale stereo audio-video dataset with paired motion tracks, and StereoWorldBench for measuring audiovisual spatial consistency. Experiments show that StereoBind substantially improves spatial alignment in stereo audio generation over existing models while preserving overall audiovisual quality.
