Search papers, labs, and topics across Lattice.
This paper introduces MemoryWAM, an efficient world action model that integrates persistent memory to enhance robotic manipulation by effectively modeling both recent and historical observations. By employing a hybrid memory design that utilizes recent frames, event-boundary anchors, and gist tokens, MemoryWAM addresses the trade-off between inference efficiency and the ability to handle non-Markovian environments. Experimental results demonstrate that MemoryWAM significantly outperforms existing vision-language-action and world action model baselines in long-horizon manipulation tasks while maintaining lower computational costs.
MemoryWAM achieves superior performance in robotic manipulation tasks by efficiently leveraging both short-term and long-term memory without sacrificing computational efficiency.
Robust robotic manipulation in the real world requires not only an understanding of the current observation, but also memory and dynamics modeling. World action models (WAMs) possess these capabilities by jointly modeling visual foresight and actions conditioned on both current and historical observations, making them a promising paradigm for robotic manipulation. However, existing WAMs face a fundamental trade-off: methods with efficient inference typically condition only on a bounded window of recent observations and therefore struggle in non-Markovian environments, whereas methods that preserve long histories incur time and space costs that grow substantially with sequence length. To address this challenge, we introduce MemoryWAM, a world action model with efficient persistent memory. MemoryWAM uses a hybrid memory design that combines recent frames, event-boundary anchor frames, and compact gist tokens that summarize long-range history. A tailored attention mechanism enables retrieval of both detailed short-term context and compressed long-term context, supporting memory-dependent decision-making with reduced inference latency and GPU memory usage. Across long-horizon, memory-dependent manipulation tasks in both simulation and the real world, MemoryWAM outperforms strong vision-language-action (VLA) and WAM baselines while maintaining favorable computational efficiency.
Tsinghua AI