Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces Binaural Difference Attention with Action Transition Prediction (BDATP) to improve generalization in Audio-Visual Navigation (AVN) tasks. BDATP uses a Binaural Difference Attention (BDA) module to model interaural differences for better spatial orientation and an Action Transition Prediction (ATP) task to regularize the policy and prevent overfitting. Experiments on Replica and Matterport3D show BDATP significantly improves Success Rates, achieving up to a 21.6% absolute improvement on the Replica dataset for unheard sounds.
Overfitting to semantic sound features is holding back audio-visual navigation, but explicitly modeling binaural differences and regularizing action transitions can boost success rates by over 20%.
In Audio-Visual Navigation (AVN), agents must locate sound sources in unseen 3D environments using visual and auditory cues. However, existing methods often struggle with generalization in unseen scenarios, as they tend to overfit to semantic sound features and specific training environments. To address these challenges, we propose the \textbf{Binaural Difference Attention with Action Transition Prediction (BDATP)} framework, which jointly optimizes perception and policy. Specifically, the \textbf{Binaural Difference Attention (BDA)} module explicitly models interaural differences to enhance spatial orientation, reducing reliance on semantic categories. Simultaneously, the \textbf{Action Transition Prediction (ATP)} task introduces an auxiliary action prediction objective as a regularization term, mitigating environment-specific overfitting. Extensive experiments on the Replica and Matterport3D datasets demonstrate that BDATP can be seamlessly integrated into various mainstream baselines, yielding consistent and significant performance gains. Notably, our framework achieves state-of-the-art Success Rates across most settings, with a remarkable absolute improvement of up to 21.6 percentage points in Replica dataset for unheard sounds. These results underscore BDATP's superior generalization capability and its robustness across diverse navigation architectures.