Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper explores a novel approach to enhance spatial audio generation in real-world speech scenes by leveraging visually guided First-Order Ambisonics (FOA). The authors introduce the YT-SPEECH dataset, specifically designed for this task, and propose a two-stage Localizer-Renderer framework that utilizes audio-visual segmentation to generate spatial heatmaps and a conditional complex-domain U-Net for reconstructing directional FOA signals. Experimental results demonstrate significant improvements in reconstruction fidelity, spatial accuracy, and perceptual speech quality compared to existing methods and ablated models.
Improved spatial audio generation in real-world speech scenes achieves unprecedented fidelity and accuracy using a visually guided framework.
Spatial audio is a key component of immersive $360^\circ$ media, yet high-quality spatial capture remains limited in real-world speech-dominant scenes. We study visually guided First-Order Ambisonics (FOA) speech spatialization in the wild: given aligned $360^\circ$ video and an omnidirectional audio track, we recover the missing directional FOA components. To support this task, we introduce YT-SPEECH, a speech-oriented $360^\circ$ video-FOA dataset curated from YouTube. We propose a two-stage Localizer-Renderer framework, where an audio-visual segmentation backbone provides frame-wise spatial heatmaps and a conditional complex-domain U-Net reconstructs directional FOA signals from the omnidirectional channel. A confidence-based gating strategy stabilizes conditioning under ambiguous acoustic conditions. Experiments show improved reconstruction fidelity, spatial accuracy, and perceptual speech quality relative to ablated variants and prior approaches.