Search papers, labs, and topics across Lattice.
This paper introduces MemoryVLA++, a novel framework for temporal modeling in vision-language-action (VLA) models that enhances robotic manipulation through the integration of memory and imagination. By leveraging a Perceptual-Cognitive Memory Bank and a world model for future state prediction, the framework allows robots to effectively utilize past interactions and anticipate future actions, addressing the limitations of traditional VLA models that primarily focus on current observations. Extensive experiments demonstrate significant performance improvements across various simulation and real-robot tasks, with notable gains of up to 28% in memory and imagination-dependent scenarios.
MemoryVLA++ achieves up to 28% performance gains in robotic manipulation tasks by integrating memory and imagination, transforming how robots handle temporal dependencies.
Temporal modeling is essential for robotic manipulation, as effective control requires both memory of past interactions and imagination of future states. However, most VLA models rely primarily on the current observation and therefore struggle with long-horizon, temporally dependent tasks. Cognitive science suggests that humans rely on working memory to buffer short-lived context, the hippocampal system to preserve episodic memory of past experience, and internal models to imagine possible future state evolution. Inspired by these mechanisms, we propose MemoryVLA++, a full temporal modeling framework that equips VLA models with memory and imagination for robotic manipulation. A pretrained VLM encodes the current observation into perceptual and cognitive tokens, forming working memory. These tokens query a Perceptual-Cognitive Memory Bank to retrieve relevant historical context. This bank stores low-level details and high-level semantics from past interactions, and is updated through redundancy-aware consolidation. A world model imagines future states in a denoising latent space, and the imagined latents are integrated under memory guidance to form full temporal-aware tokens. The resulting tokens condition a diffusion action expert to predict temporally consistent action sequences. We conduct extensive experiments on 5 simulation benchmarks and 3 categories of real-robot tasks across 3 robots, covering general manipulation, long-horizon temporal tasks, robustness, and generalization. Our method achieves strong performance across Libero, SimplerEnv, Mikasa-Robo, Calvin, Libero-Plus, and diverse real-robot tasks, validating the effectiveness of full temporal modeling with memory and imagination. For example, on real robots, it achieves +9%, +26%, +28% gains on general, memory-dependent, and imagination-dependent tasks. Project Page: https://shihao1895.github.io/MemoryVLA-PP-Web
