Search papers, labs, and topics across Lattice.
This paper introduces ST-OmniQA, a novel spatio-temporal audio-visual question-answering benchmark designed to evaluate the ability of models to reason about dynamic sound sources in relation to their spatial and temporal characteristics. The authors present ST-Omni-R1, which combines first-order Ambisonics audio with panoramic visual context, utilizing progressive curriculum learning and reasoning-tree reinforcement learning to enhance model performance. ST-Omni-R1 achieves a significant average semantic accuracy of 77.83% across various reasoning tasks, outperforming the best baseline by a substantial margin, demonstrating its effectiveness in sound-event recognition and localization.
ST-Omni-R1 not only excels in sound-event recognition but also sets a new standard for spatial audio reasoning, achieving nearly double the accuracy of existing models.
Understanding dynamic sound sources requires jointly determining what produces a sound, where the source is located, and how it moves over time. Yet existing audio-language models often represent clips as global acoustic events, while vision-language models lack the spatial audio cues needed to localize and track individual sources. To evaluate this missing capability, we introduce ST-OmniQA, a spatio-temporal audio-visual question-answering benchmark built from panoramic videos paired with synchronized first-order Ambisonics (FOA) audio of moving sound sources. It contains 40K videos and 400K question-answer pairs organized into four capability levels covering sound-event recognition, direction of arrival, source distance, motion trajectories, and temporally grounded audio-visual reasoning. Building on this benchmark, we propose ST-Omni-R1, which integrates FOA-derived semantic and trajectory representations with panoramic visual context and is trained through progressive curriculum learning and reasoning-tree reinforcement learning. ST-Omni-R1 achieves 77.83\% average semantic accuracy across the four levels, compared with 37.28\% for the best evaluated baseline. Results on three public spatial-audio benchmarks further indicate that its learned spatial and motion representations transfer beyond ST-OmniQA.
NUS