Search papers, labs, and topics across Lattice.
This paper introduces SG-Ego, a large-scale annotation set that enhances Ego4D with spatio-temporal scene graphs to facilitate structured reasoning about human activities in first-person videos. The authors propose GLEN, a graph-based model that processes sequences of scene graphs to align them with textual actions and predict their temporal evolution, addressing the novel activity-driven graph-edit forecasting (A-GEF) problem. The results show that GLEN outperforms traditional video baselines and excels in long-horizon reasoning tasks, highlighting the effectiveness of explicit, compositional representations for understanding dynamic scenes.
GLEN not only outperforms traditional video models but also enables structured, interpretable predictions of how scenes evolve with human activities.
Understanding human behavior while interacting with the surrounding world is crucial for many applications of embodied AI. First-person videos are particularly informative for this problem, as they well capture how activities reshape the scene over time. However, existing approaches often rely on implicit visual or language-aligned representations, disregarding structured reasoning over the scene dynamic. We argue that explicit, compositional and editable representations of human-environment interactions can play a crucial role for rich grounded activity understanding. To this end, we introduce SG-Ego, a large scale annotation set extending Ego4D with spatio-temporal scene graphs, where relations triplets are consolidated over time into explicit time-evolving descriptions of the scene state. To reason over this representation, we propose GLEN, a graph-based model that operates over scene graph sequences to both align them with textual actions and model their temporal evolution. In addition, we formulate the activity-driven graph-edit forecasting (A-GEF) problem, a novel task that casts scene dynamics as a sequence of structured transformations conditioned on ongoing actions, enabling explicit reasoning about how scenes change over time. We validate our approach across multiple downstream tasks, spanning retrieval benchmarks as EgoMCQ and EgoCVR, as well as long-horizon reasoning benchmarks as EXPLORE-Bench and the newly introduced A-GEF. GLEN achieves strong results compared to raw video baselines and it excels in reasoning settings, typically addressed only with MLLMs, while enabling controllable and structured predictions of scene dynamics driven by human activities. We believe our results establish spatio-temporal scene graphs, together with models that reason over them, as strong compositional and interpretable representations for video understanding and potentially beyond.
