Search papers, labs, and topics across Lattice.
This paper introduces LongEgoRefer, a benchmark specifically designed for Video Referring Expression Comprehension (Video REC) in long-form egocentric videos, addressing the limitations of existing benchmarks that focus on short clips. The benchmark consists of 1,498 referring expressions across videos averaging 45 minutes in length, characterized by sparse object occurrences and complex human-object interactions. Evaluation of current Video REC models reveals significant challenges in spatio-temporal grounding, underscoring the need for improved video understanding capabilities in real-world scenarios.
Existing Video REC models falter dramatically when faced with the complexities of long-form egocentric videos, revealing a critical gap in current methodologies.
Egocentric videos capture rich and diverse human-object interactions and have emerged as a fundamental resource for understanding human activities related to objects. In this context, Video Referring Expression Comprehension (Video REC), the task of localizing the temporal and spatial extent of a referred object in video frames given a natural language query, plays a key role in linking textual descriptions to observed objects in untrimmed egocentric recordings. However, existing egocentric Video REC benchmarks primarily focus on short video clips, where some target object appears densely within frames. Such settings do not reflect real-world egocentric recordings, which are long-form, untrimmed, and characterized by sparse object occurrences and complex activity transitions. To address this limitation, we introduce LongEgoRefer, a novel and challenging benchmark constructed from long-form videos in the Ego4D dataset. LongEgoRefer contains 1,498 referring expressions with an average video duration of 45 minutes. The benchmark exhibits extreme target sparsity, detailed linguistic descriptions, and complex human-object interactions embedded in long, dynamic egocentric narratives. Consequently, it defines a demanding spatio-temporal grounding problem that requires models to identify both when an event occurs and where the referred object appears within extended video sequences. We evaluate existing Video REC approaches, including training-free baselines based on vision-language models combined with Grounded SAM2. Extensive experiments show that even advanced baselines and current state-of-the-art models struggle significantly on LongEgoRefer. These results highlight the intrinsic difficulty of long-form egocentric spatio-temporal grounding and emphasize the need for more robust video understanding models.
