Search papers, labs, and topics across Lattice.
This paper introduces Reversal Q-Learning (RQL), a novel off-policy reinforcement learning algorithm that leverages iterative generative modeling techniques to enhance offline RL performance. By treating flow refinement steps as distinct actions within an expanded Markov decision process, RQL generates virtual on-policy trajectories through flow reversal and employs bias-and-variance reduction to address the curse of horizon. Experimental results demonstrate that RQL outperforms existing flow-based offline RL methods across 50 challenging simulated robotic tasks, highlighting its effectiveness in utilizing prior data more efficiently.
RQL achieves superior offline reinforcement learning performance by effectively utilizing prior data without the pitfalls of backpropagation through time.
Iterative generative modeling techniques, such as flow matching, provide powerful tools to model complex behaviors for effective offline reinforcement learning (RL). In this work, we propose a new off-policy RL algorithm that trains a flow policy based on prior data. Our idea starts from the "expanded" Markov decision process (MDP) framework, which treats individual flow refinement steps as separate actions in an MDP. To enable off-policy RL within this framework, we apply two techniques: we generate virtual on-policy trajectories (by "reversing" flows) to make this framework compatible with prior data, and we apply a bias-and-variance reduction technique to mitigate the curse of horizon in off-policy RL. We call the resulting algorithm Reversal Q-learning (RQL). RQL has several advantages over previous flow-based RL methods: it does not suffer from backpropagation through time, makes better use of the learned value function, and directly trains the full, expressive flow policy. Through our experiments on 50 challenging simulated robotic tasks, we show that RQL leads to the best average offline RL performance compared to state-of-the-art flow-based offline RL algorithms.
BAIR