Search papers, labs, and topics across Lattice.
The Action-Effect Memory (AEM) pretraining framework enhances robot manipulation by learning compact temporal representations from vision-action histories, addressing the limitations of single-frame visual encoding. By modeling manipulation as an action-driven interaction process and employing masked modeling to recover missing content, AEM effectively captures action-conditioned state evolution. Evaluations demonstrate that AEM significantly improves performance across various manipulation tasks in both simulated and real-world environments, outperforming traditional baselines while reducing inference latency and computational costs.
AEM's innovative approach to learning from incomplete histories leads to superior manipulation performance, even in complex and cluttered environments.
We present AEM, an Action-Effect Memory pretraining framework for robot manipulation that learns compact temporal representations from vision-action history. Unlike prior robot representation pretraining methods that mainly focus on single-frame visual encoding, AEM targets the temporal nature of manipulation, where the current observation alone is often insufficient under partial observability. AEM models manipulation as an action-driven interaction process by interleaving visual and action features and applying masked modeling to recover missing content from incomplete histories, thereby learning action-conditioned state evolution. The Mamba-encoded output of the final vision token is used as a compact history representation, serving as the global context for decoding and downstream control. This design preserves a single-vector temporal bottleneck while keeping inference efficient. We evaluate AEM with Diffusion Policy and Flow Policy. AEM consistently improves manipulation performance in both simulation and real-world settings, outperforming baselines across clean scenes, cluttered and random scenes, and non-Markovian tasks. Ablation studies further show that history-aware pretraining surpasses single-frame pretraining and direct frame stacking, while reducing inference latency and computational cost.
