Search papers, labs, and topics across Lattice.
This paper introduces FlowDPG, a novel DDPG-style method tailored for flow matching policies in robotic manipulation, which overcomes the limitations of backpropagating through time (BPTT) by distilling the critic gradient into the velocity field during training. By integrating demonstration-driven and critic-driven components, FlowDPG achieves stable policy improvement without the computational burden of BPTT. In real-world experiments, FlowDPG demonstrated a remarkable 92% success rate on a complex dual-arm AirPods assembly task, significantly outperforming existing reinforcement learning approaches.
FlowDPG achieves a 92% success rate in real-world robotic manipulation, bypassing the computational pitfalls of traditional policy gradient methods.
Real-world reinforcement learning for robotic manipulation remains challenging, and this difficulty is amplified for flow matching policies: applying policy gradient methods to these policies is fundamentally limited by the need to backpropagate through time(BPTT) along the multi-step ODE that maps noise to actions, which is computationally prohibitive and numerically fragile. We propose FlowDPG, a DDPG-style method specifically designed for flow matching policies that distills the critic gradient into the velocity field at training time, bypassing BPTT entirely. Intuitively, FlowDPG combines two complementary vectors: the demonstration-driven velocity that keeps the action feasible, and the critic-driven correction that steers it toward higher value. Our contributions are threefold: (1) a BPTT-free distillation framework that enables stable DDPG-style policy improvement on flow matching policies, (2) a formal connection between the FlowDPG update direction and vanilla Deterministic Policy Gradient via three explicit approximations, and (3) real-world validation on a long-horizon, multi-stage, dual-arm AirPods assembly task, where FlowDPG attains a 92% end-to-end success rate, substantially outperforming recent RL methods spanning value-conditioning, auxiliary-module adaptation, and adjoint-based critic-gradient approaches. Videos and more results are provided on the project page https://flowdpg.github.io.
CMU ML