Search papers, labs, and topics across Lattice.
This paper introduces a novel framework for egocentric video question answering (VQA) that utilizes Egocentric Scene Graphs (EgoSGs) to address the token limitations faced by multi-modal large language models (MLLMs) when processing long video sequences. By transforming videos into structured, temporally grounded representations, the method significantly reduces input length while preserving essential visual and temporal information, enabling efficient reasoning over entire video sequences. The proposed approach achieves state-of-the-art performance on the HD-EPIC VQA dataset, outperforming existing video-based models and highlighting the potential of structured representations in enhancing video understanding.
Transforming long-form videos into compact, temporally grounded scene graphs allows MLLMs to maintain semantic richness while overcoming input constraints, leading to state-of-the-art VQA performance.
Existing multi-modal large language models (MLLMs) face significant challenges in processing long video sequences due to strict input token limitations. As a result, current video understanding approaches, especially in egocentric settings characterized by complex dynamics, frequent state changes, and moving cameras, are forced to massively subsample frames. This leads to severe loss of temporal and contextual information, constraining their ability to perform fine-grained video reasoning. In this work, we introduce a framework for egocentric video question answering (VQA) that overcomes these input constraints through Egocentric Scene Graphs (EgoSGs), i.e., temporally grounded, structured representations that capture objects, attributes, spatial relations, and interactions over time. By representing videos as compact, text-based scene graphs, our method preserves the essential visual and temporal information of the original video in a symbolic form that drastically reduces input length while maintaining semantic richness. Crucially, this enables MLLMs to reason efficiently over entire video sequences within their token budget. On HD-EPIC VQA, our method achieves state-of-the-art results, outperforming strong video-based baselines on multiple models and suggesting that structured, temporally grounded representations like EgoSGs can bridge long-form egocentric video understanding and the context limitations of today's MLLMs.
