Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces Ledger, a persistent 3D object memory system designed for embodied agents to recall objects from egocentric video streams, even those not directly interacted with. Ledger achieves this by clustering object observations by resting locations, recording moves only after repeated evidence, and augmenting spatial data with contextual descriptions. The system significantly improves accuracy on downstream tasks like HD-EPIC and UCS-Bench, demonstrating the value of temporal persistence and contextual information for robust 3D object memory.
Embodied agents can now remember objects they never touched by tracking their resting locations and inferring moves from repeated evidence, boosting downstream task accuracy by over 10%.
As we move through the world and carry out everyday tasks, we encounter objects that may become relevant only later. We are capable of recalling where we left something or what was inside a container, even without knowing we would need it later. Here, we study how an embodied assistant can build a similar memory from egocentric videos, by observing a person's day-to-day activities. We present Ledger, a persistent 3D object memory that combines object locations, their histories, and contextual descriptions. It associates observations across the recording and retains objects after they leave the view, including those the person never touches. It clusters each object's observations by resting locations and records a move only after repeated evidence, reducing the effect of localization noise. Short descriptions preserve details such as an object's contents or supporting surface. It saves these records to later answer spatial questions without having to access the original images or video. Our memory raises HD-EPIC accuracy from 29.7% to 42.6%, UCS-Bench accuracy from 33.8% to 38.5% and localizes Ego4D objects with a 0.99 m median error on returned predictions. Our analyses identify complementary roles for temporal persistence, contextual descriptions, and retrieval. Our study on 100 stitched streams of multiple scenes each further exposes failures in both retrieval and construction. Per-scene construction partially recovers the performance lost across scene changes compared to that of single scene streams.
Johns Hopkins