Track manipulation, locomotion and embodied systems.
Search papers, labs, and topics across Lattice.
This paper introduces an agentic framework for proactive robots that leverages a Procedural Assistance Memory (PAM) to infer user intentions and provide timely assistance. PAM, structured as a finite-state machine, is dynamically updated by a reflection VLM based on daily interaction traces, enabling the robot to learn recurring routine patterns and reduce ambiguity in proactive decisions. The proposed system significantly outperforms memory-free baselines in a desk-assistance evaluation, demonstrating improved precision, recall, F1, and reduced first-fire delay, while also adapting to routine changes through continuous memory reflection.
Robots can learn to proactively assist users by dynamically updating a procedural memory, dramatically reducing assistance latency and improving task recall over time.
Proactive robots should anticipate user needs and provide assistance before being explicitly asked. A straightforward approach is to infer human intentions from commonsense knowledge, but the same observed behavior can imply different needs across users, potentially leading to unwanted assistance. We observe that everyday routines in the real world exhibit recurring patterns, so learning these patterns can reduce ambiguity in proactive decisions. Based on this, we propose a memory-based agentic framework in which a VLM task predictor reads a Procedural Assistance Memory (PAM) alongside the current observation to decide when to assist and what task to perform, and a VLA executor performs the predicted task. PAM is structured as a finite-state machine over activity phases, per-state assistance rules, and event-driven transitions; a reflection VLM revises PAM from each day's interaction trace without weight updates or retraining. In our 5-day desk-assistance evaluation, memory-free baselines remain conservative, with limited recall on proactive tasks. Our system improves consistently with accumulated experience, achieving the highest precision, recall, and F1 and reducing first-fire delay from approximately 5 seconds on Day 1 to 1 second by Day 5. Our approach also adapts to changes in the user's routine through continued memory reflection, recovering from memory--routine misalignment after a single memory update. The project page is available at https://davian-robotics.github.io/PAM/