Search papers, labs, and topics across Lattice.
This paper introduces UniMem, a novel framework that integrates multimodal memory and control within a single backbone for Vision-Language-Action (VLA) models, addressing the limitations of existing memory management approaches. By employing an event classifier for memory updates and a keyframe caching technique, UniMem significantly enhances performance on non-Markovian tasks while reducing the complexity of the training pipeline. Experimental results show that UniMem outperforms traditional fixed-interval image sampling and hierarchical baselines, achieving 93.4% accuracy in simulation tasks and 80.0% in hardware tasks, alongside faster inference times.
UniMem achieves a remarkable 93.4% accuracy on long-horizon tasks by unifying multimodal memory and control, outperforming conventional methods that rely on fixed-interval sampling.
While Vision-Language-Action (VLA) models have leveraged internet-scale pretraining and task-focused finetuning to achieve strong performance on long-horizon tasks, they often struggle with non-Markovian tasks that require memory. Existing approaches to memory typically involve additional Vision-Language-Models (VLMs) for long-term memory management, introducing a memory bottleneck and a fractured training pipeline. Conditioning on multiple historical frames can provide the VLA with access to more descriptive features of past scenes, but can degrade performance if frames are chosen at arbitrary, fixed intervals. To address these limitations, we present UniMem, a framework that unifies high-level, multimodal memory and low-level control under one backbone. UniMem employs an event classifier for memory updates, a keyframe encoder for dense spatial memory, and a keyframe caching technique to minimize overhead during policy rollouts. We evaluate UniMem across five simulation and four hardware tasks targeting sequential and spatial memory, demonstrating that our unified, single-model system outperforms fixed-interval image sampling baselines (93.4% vs. 68.2%) in simulation and hierarchical baselines (80.0% vs. 43.5%) in hardware, while offering faster inference and a simple training pipeline for easy adoption. Project website: https://losterberg3.github.io/unimem-vla/
