Search papers, labs, and topics across Lattice.
QF3 is a novel off-policy RL algorithm designed to accelerate the training of flow policies for robotic control by leveraging filtered Q-gradients. It addresses the challenge of unreliable critic updates by restricting gradient application to action dimensions close to the replay action, enabling efficient learning from interaction. QF3 achieves significant speedups and enables zero-shot hardware transfer for humanoid locomotion, outperforming existing on-policy methods.
Humanoid locomotion learned from scratch and transferred to hardware in 10x less time, thanks to a novel off-policy flow RL approach.
Flow policies have become a standard policy class for learning robot behaviors from demonstrations, but reinforcement learning is still critical for improving pre-trained flow policies or learning them from scratch through interaction. We introduce QF3 (Fast Flow RL with Filtered Q-Gradients), an online off-policy RL algorithm that trains a flow policy with flow matching plus the critic's action gradient, backpropagated through a one-step prediction of the flow's output. To keep updates where the critic and this prediction are reliable, QF3 applies the critic gradient only to action dimensions that stay near the replay action. To our knowledge, QF3 is the first off-policy flow RL method to train humanoid locomotion policies from scratch and transfer them zero-shot to hardware. Paired with a high-throughput off-policy training recipe, it trains humanoid locomotion and motion-tracking policies with a 10x wall-clock speedup over FPO++, a recent on-policy flow RL method. We further apply QF3 to fine-tune pretrained flow-based manipulation policies on both ABC-Sim and Robomimic tasks. These results suggest that QF3 can both learn robot policies from scratch and refine those acquired from demonstrations. Website: https://qf3-rl.github.io/
BAIR