Search papers, labs, and topics across Lattice.
The paper introduces EgoSound, a new benchmark for evaluating egocentric sound understanding in Multimodal Large Language Models (MLLMs), leveraging data from Ego4D and EgoBlind. EgoSound comprises a seven-task taxonomy, including intrinsic sound perception, spatial localization, causal inference, and cross-modal reasoning, with 7315 QA pairs across 900 videos. Experiments on nine state-of-the-art MLLMs using EgoSound reveal limitations in fine-grained spatial and causal auditory reasoning, despite exhibiting emerging auditory abilities.
MLLMs can "hear" a little, but EgoSound reveals they're still largely deaf to the nuances of sound in egocentric video, especially when it comes to spatial and causal reasoning.
Multimodal Large Language Models (MLLMs) have recently achieved remarkable progress in vision-language understanding. Yet, human perception is inherently multisensory, integrating sight, sound, and motion to reason about the world. Among these modalities, sound provides indispensable cues about spatial layout, off-screen events, and causal interactions, particularly in egocentric settings where auditory and visual signals are tightly coupled. To this end, we introduce EgoSound, the first benchmark designed to systematically evaluate egocentric sound understanding in MLLMs. EgoSound unifies data from Ego4D and EgoBlind, encompassing both sighted and sound-dependent experiences. It defines a seven-task taxonomy spanning intrinsic sound perception, spatial localization, causal inference, and cross-modal reasoning. Constructed through a multi-stage auto-generative pipeline, EgoSound contains 7315 validated QA pairs across 900 videos. Comprehensive experiments on nine state-of-the-art MLLMs reveal that current models exhibit emerging auditory reasoning abilities but remain limited in fine-grained spatial and causal understanding. EgoSound establishes a challenging foundation for advancing multisensory egocentric intelligence, bridging the gap between seeing and truly hearing the world.
