Search papers, labs, and topics across Lattice.
This paper introduces GESTO, a novel spatio-temporal memory system that integrates a persistent 4D scene graph with a hierarchical representation of human-object interactions and goal-driven events. By processing RGB-D observation streams, GESTO effectively extracts and grounds timestamped interactions, enabling robust activity-centric reasoning in dynamic environments. The system demonstrates high performance on established benchmarks and new queries, indicating its ability to outperform existing methods even when deprived of ground-truth inputs.
GESTO achieves near-ground-truth performance in reasoning about human activities in dynamic scenes, revealing the power of hierarchical memory structures in robotic perception.
Robots operating in human environments need memories that capture not only what objects exist and where, but also how people use them over time and how individual interactions compose into goal-directed activities. Existing 4D scene graphs preserve object and place histories but omit activity structure, whereas activity representations are either not grounded in persistent 3D scenes or rely on externally provided event boundaries and object associations. We present GESTO (Grounded Event and Spatio-Temporal memOry), a spatio-temporal memory that couples a persistent 4D scene graph with a two-level hierarchy of atomic human--object interactions and goal-driven events. From an RGB-D observation stream, GESTO automatically extracts timestamped interactions, grounds them to persistent scene entities, groups them into events, and uses event context to refine uncertain object associations. A relation-aware tool-calling agent queries the resulting memory for activity-centric spatio-temporal reasoning. We evaluate GESTO on the reproducible text, binary, and time categories of an existing benchmark, together with 40 new Space2Event and Event2Space queries. GESTO achieves scores of 0.71, 0.75, and 0.70 on the standard categories, approaching a method supplied with ground-truth event and object grounding, while substantially outperforming the same reasoning framework when these inputs are removed. It further achieves 0.73 and 0.75 on Space2Event and Event2Space queries. Ablations show that hierarchical event structure and context-aware grounding refinement provide complementary benefits, supporting activity-grounded hierarchical memory for retrospective reasoning in dynamic human environments.
Google Research