Search papers, labs, and topics across Lattice.
This paper introduces a framework for online robot policy refinement using Vision-Language Models (VLMs) as reward generators. A VLM is trained on a large, multi-source dataset and used to create a multifaceted reward signal based on visual observations, encompassing process, completion, and temporal contrastive rewards. The approach demonstrates significant improvement in success rates on long-horizon manipulation tasks within just 30 RL iterations, showcasing the potential of VLMs to replace manual reward engineering in robot learning.
Forget hand-engineered reward functions: this work shows VLMs can provide reliable, zero-shot feedback for online robot policy refinement, boosting success rates on manipulation tasks in just 30 RL iterations.
Reinforcement Learning (RL) has shown great potential in refining robotic manipulation policies, yet its efficacy remains strongly bottlenecked by the difficulty of designing generalizable reward functions. In this paper, we propose a framework for online policy refinement by adapting foundation VLMs into online reward generators. We develop a robust, scalable reward model based on a state-of-the-art VLM, trained on a large-scale, multi-source dataset encompassing real-world robot trajectories, human-object interactions, and diverse simulated environments. Unlike prior approaches that evaluate entire trajectories post-hoc, our method leverages the VLM to formulate a multifaceted reward signal comprising process, completion, and temporal contrastive rewards based on current visual observations. Initializing with a base policy trained via Imitation Learning (IL), we employ these VLM rewards to guide the model to correct sub-optimal behaviors in a closed-loop manner. We evaluate our framework on challenging long-horizon manipulation benchmarks requiring sequential execution and precise control. Crucially, our reward model operates in a purely zero-shot manner within these test environments. Experimental results demonstrate that our method significantly improves the success rate of the initial IL policy within just 30 RL iterations, demonstrating remarkable sample efficiency. This empirical evidence highlights that VLM-generated signals can provide reliable feedback to resolve execution errors, effectively eliminating the need for manual reward engineering and facilitating efficient online refinement for robot learning.
