Search papers, labs, and topics across Lattice.
This paper introduces HyMeS, a hybrid learning framework that enhances memory-dependent robot manipulation by combining low-level motor skill learning with high-level memory management. By utilizing gradient-based imitation learning for motor skills and heuristic learning for memory strategies, HyMeS effectively addresses the non-Markovian nature of real-world tasks. The framework demonstrates significant improvements in task performance on RoboMemArena, achieving a mean cumulative success rate of 66.2% and a mean task success rate of 60.1%, marking a notable advancement over existing methods.
HyMeS enables robots to efficiently manage long-horizon interactions, achieving a 14.5-point increase in task success without requiring extensive demonstrations for every task configuration.
Modern vision-language-action (VLA) policies have acquired broad manipulation skills, but typically generate each action chunk from the current observation or a short fixed-length history. However, real-world manipulation is often non-Markovian, requiring robots to retain and reason over task-relevant information from long-horizon interaction histories to determine the next action. To address this challenge, we propose HyMeS, a hybrid learning framework that leverages the reasoning and memory-management capabilities of coding agents to steer a Markovian VLA for memory-dependent manipulation. Specifically, HyMeS learns low-level motor skills through gradient-based imitation learning, while a coding agent acquires high-level memory-management strategies through heuristic learning by iteratively updating an executable heuristic system from rollout feedback. Furthermore, we close the loop between steering and execution through multimodal stage-completion verification, which updates memory using proprioceptive signals and multi-frame VLM judgments. Compared with end-to-end memory-augmented VLAs, HyMeS requires demonstrations only for reusable motor skills rather than for every history-dependent task configuration, enabling data-efficient compositional generalization. On RoboMemArena, HyMeS improves mean cumulative success from 52.5% to 66.2% and mean task success from 41.3% to 60.1% over pi0.5, while outperforming PrediMem by 4.5 points in cumulative success and 14.5 points in task success.
Stanford HAI