Search papers, labs, and topics across Lattice.
This survey critically reviews the application of vision-language models (VLMs) in egocentric video understanding, highlighting their potential to enhance wearable intelligence and human-robot interaction. It addresses the unique challenges posed by egocentric perspectives, such as ego-motion and occlusion, while organizing existing literature around key tasks and methodologies including hand-object interaction and temporal reasoning. A major finding is that while VLMs excel in recognizing visible objects, they struggle with understanding evolving interactions and user intent over extended activities, underscoring the need for improved interaction-aware supervision and temporally grounded reasoning.
Current vision-language models excel at recognizing objects but falter in capturing dynamic interactions and user intent over time, revealing critical gaps in embodied AI.
Egocentric video captures activities from the wearer's perspective, providing a direct view of human attention, hand--object interaction, and goal-directed behavior. This perspective is increasingly important for wearable intelligence, assistive systems, human--robot interaction, and embodied AI, yet it introduces challenges including ego-motion, occlusion, small active objects, viewpoint-dependent appearance, and long-range temporal dependencies. Vision--language models (VLMs) offer a promising foundation for addressing these challenges by linking visual observations with semantic knowledge and natural-language supervision. This survey presents a critical review of VLMs for egocentric video understanding, tracing the progression from conventional recognition architectures to multimodal foundation models and embodied systems. We organize the literature around tasks, datasets, hand--object interaction understanding, temporal reasoning, frame and clip selection, multimodal representation learning, prompting, semantic alignment, and model adaptation. Particular attention is given to graph-based and object-centric reasoning as mechanisms for modeling relations among hands, objects, actions, and scene context over time. We further examine how first-person perception and multimodal foundation models support wearable assistance, robot skill learning, human-to-robot transfer, and embodied decision making. Across the reviewed literature, a consistent limitation emerges: current models recognize visible objects more reliably than evolving interactions, actions, and user intent, especially over long activities. We therefore identify temporally grounded reasoning, interaction-aware supervision, efficient long-video processing, multimodal fusion, graph-enhanced representations, cross-domain generalization, privacy, and trustworthy evaluation as key priorities for deployable embodied intelligence.
