Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces Optimal Transport Q-Learning (OTQL), a novel method designed to fine-tune and accelerate flow-based policies by leveraging the robot's own experience through reinforcement learning post-training. The approach addresses the challenges of suboptimal policy behaviors and distribution shifts by employing advantage weighted conditional optimal transport flow matching, allowing for effective policy improvement with a limited interaction budget. Experimental results demonstrate that OTQL significantly enhances the success rates of single-task policies and pre-trained vision language action models while drastically reducing inference steps, showcasing its efficiency in both simulation and real-world applications.
OTQL boosts the success rates of robotic policies from 36% to 86% while slashing inference steps by 70%, revolutionizing flow policy optimization.
Diffusion and flow policies have recently demonstrated remarkable performance in robotic applications by accurately capturing multimodal robot trajectory distributions, especially in the context of vision language action (VLA) models. However, high quality policy performance also requires fast inference and high quality demonstrations, which are often hard to get. Lack of these leads to suboptimal policy behaviors and failure under distribution shifts. In this work we address the problem of fine-tuning and accelerating suboptimal flow-based policies using the robot's experience through RL post-training. We introduce Optimal Transport Q-Learning (OTQL), a new method for finetuning flow policies using advantage weighted conditional optimal transport flow matching. OTQL can finetune and accelerate flows with an interaction budget of 50-60 episodes while avoiding computationally expensive distillation in simulation and real-world robot tasks. Our results show that OTQL post-trains flow policies using the robot's own experience, increasing average success percentage of single-task policies from 36% to 86% and of a pre-trained VLA from 38% to 76% while reducing the number of inference steps per action generation by 70%.