Search papers, labs, and topics across Lattice.
This paper introduces a retrieval-augmented approach to extend vision-language-action (VLA) policies to new tasks without the need for costly task-specific fine-tuning. By training a single policy on demonstrations from a target embodiment and a cheaper embodiment, the authors show that new tasks can be integrated at deployment by appending additional demonstrations to a retrieval pool, allowing the policy to adapt dynamically. The results indicate that this method not only enhances performance across various VLA models but is particularly effective in the Cosmos Policy framework, where it leverages visual consistency to improve action quality.
Retrieval can replace costly fine-tuning in VLA models, enabling seamless adaptation to new tasks with just a single trained policy.
Extending a vision-language-action (VLA) policy to a new task typically requires task-specific teleoperated demonstrations and per-task fine-tuning, making adaptation costly in both data collection and compute. In this paper, we show that this target-side per-task adaptation cost can be replaced by retrieval. Our retrieval-augmented policy is trained once on paired demonstrations from the target embodiment (query) and a cheaper embodiment (pool, e.g., human-hand video), then frozen. New tasks are added at deployment by appending pool-side demonstrations to a retrieval pool. The frozen policy conditions on retrieved trajectories at every control step, so new tasks are absorbed by indexing data rather than updating parameters. Fine-tuning is needed only to take on a new, unseen embodiment, not for each new task. We show that retrieval improves policies beyond a specific backbone, including standard VLA policies, but its effect is especially pronounced in Cosmos Policy, a video-generation-based world-action model (WAM). In this setting, retrieval supplies coarse task progression, while the WAM's future-image objective provides an additional visual consistency signal that strengthens the retrieval-conditioned actions. On PushT, we study how retrieval provides a reusable high-level motion prior for cross-embodiment generalization to unseen goal angles, while on RoboTwin 2.0 our method outperforms cross-embodiment baselines on unseen tasks, and we additionally demonstrate the method on a real robot.
