Search papers, labs, and topics across Lattice.
This paper introduces Q2RL, a novel offline-to-online reinforcement learning algorithm that leverages Behavior Cloning (BC) to initialize a Q-function and then uses Q-gating to balance exploration with exploitation. Q2RL extracts a Q-function from a BC policy through limited environment interaction and then uses Q-values to selectively switch between BC and RL policies during online training. Experiments on D4RL, robomimic, and real-world robot tasks demonstrate that Q2RL significantly outperforms state-of-the-art offline-to-online methods, achieving up to 100% success rates and 3.75x improvement over the original BC policy in just 1-2 hours of online interaction.
Forget brittle imitation learning: Q2RL unlocks robust on-robot reinforcement learning by distilling a Q-function from Behavior Cloning and intelligently gating between imitation and RL based on Q-value estimates.
Behavior Cloning (BC) has emerged as a highly effective paradigm for robot learning. However, BC lacks a self-guided mechanism for online improvement after demonstrations have been collected. Existing offline-to-online learning methods often cause policies to replace previously learned good actions due to a distribution mismatch between offline data and online learning. In this work, we propose Q2RL, Q-Estimation and Q-Gating from BC for Reinforcement Learning, an algorithm for efficient offline-to-online learning. Our method consists of two parts: (1) Q-Estimation extracts a Q-function from a BC policy using a few interaction steps with the environment, followed by online RL with (2) Q-Gating, which switches between BC and RL policy actions based on their respective Q-values to collect samples for RL policy training. Across manipulation tasks from D4RL and robomimic benchmarks, Q2RL outperforms SOTA offline-to-online learning baselines on success rate and time to convergence. Q2RL is efficient enough to be applied in an on-robot RL setting, learning robust policies for contact-rich and high precision manipulation tasks such as pipe assembly and kitting, in 1-2 hours of online interaction, achieving success rates of up to 100% and up to 3.75x improvement against the original BC policy. Code and video are available at https://pages.rai-inst.com/q2rl_website/
