Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces SIREN, a novel framework for converting monaural audio to binaural audio using visual information. SIREN employs a ViT-based encoder with dual-head self-attention to generate a shared scene map and perform end-to-end left/right channel attention, replacing traditional hand-crafted methods. The framework also incorporates a soft spatial prior and a two-stage, confidence-weighted waveform-domain fusion to minimize crosstalk, achieving improved performance on standard binaural audio datasets.
Turn monaural videos into immersive binaural experiences with SIREN, a visually guided framework that learns spatial audio cues without task-specific annotations.
Binaural audio delivers spatial cues essential for immersion, yet most consumer videos are monaural due to capture constraints. We introduce SIREN, a visually guided mono to binaural framework that explicitly predicts left and right channels. A ViT-based encoder learns dual-head self-attention to produce a shared scene map and end-to-end L/R attention, replacing hand-crafted masks. A soft, annealed spatial prior gently biases early L/R grounding, and a two-stage, confidence-weighted waveform-domain fusion (guided by mono reconstruction and interaural phase consistency) suppresses crosstalk when aggregating multi-crop and overlapping windows. Evaluated on FAIR-Play and MUSIC-Stereo, SIREN yields consistent gains on time-frequency and phase-sensitive metrics with competitive SNR. The design is modular and generic, requires no task-specific annotations, and integrates with standard audio-visual pipelines.