Search papers, labs, and topics across Lattice.
This paper introduces HitMem, a hierarchical temporal 3D memory framework designed to enhance the performance of embodied agents in dynamic environments by integrating a multi-modal context-aware retrieval mechanism. The framework addresses the limitations of traditional 3D memory systems that struggle with memory-observation conflicts due to object displacement, providing a solution that combines semantic and spatial information in a lightweight topological graph. Extensive evaluations on the Dyna-THOR benchmark show that HitMem significantly improves object relocation accuracy and reduces exploration costs, leading to better task execution performance.
HitMem revolutionizes 3D memory for embodied agents by effectively managing dynamic changes in environments, achieving remarkable improvements in task execution accuracy and efficiency.
Executing long-term tasks in dynamic environments requires embodied agents to maintain robust and adaptive 3D scene representations. However, most existing 3D memory frameworks rely on static world assumptions. When objects are displaced by human activities or unobserved events, agents encounter memory-observation conflicts and often require costly geometric recomputations or inefficient global re-exploration. To address this, we propose HitMem, a hierarchical temporal 3D memory framework with a multi-modal context-aware retrieval mechanism. Through continuous perception, HitMem unifies semantic and spatial information into a lightweight topological graph that captures support relationships, while a temporal decay mechanism dynamically regulates memory activeness to mitigate the impact of stale representations. In addition, the multi-modal context-aware retrieval mechanism defaults to filtering candidates using integrated semantic, spatial, and temporal memory features, and activates a specialized two-stage retrieval process when object displacement is detected. This process combines spatial constraints inferred from external agent trajectories with semantic common sense grounded in class affinities, efficiently identifying high-probability candidate regions. Extensive evaluations on our constructed Dyna-THOR benchmark demonstrate that HitMem significantly improves object relocation accuracy, reduces exploration costs, and enhances task execution performance in dynamic environments.
