Search papers, labs, and topics across Lattice.
This paper introduces a unified algorithmic framework for hybrid reinforcement learning in tabular MDPs, addressing the challenge of integrating offline data from environments with shifted transition dynamics. By developing two algorithms, MIN-UCB-VI for regret minimization and MAX-LCB-VI for best policy identification, the authors effectively exploit historical data while providing theoretical guarantees on performance. Extensive experiments validate the framework's optimality, revealing significant improvements in policy learning under challenging conditions.
Offline data from outdated environments can still yield optimal policies when integrated correctly, as shown by our new hybrid reinforcement learning framework.
This paper investigates a hybrid reinforcement learning setting in tabular Markov Decision Processes (MDPs), where an agent aims to learn an optimal policy by combining online interactions with a target environment and offline data from a source environment. A central challenge is that offline data may be collected from outdated environments with shifted transition dynamics, making naive integration of historical data ineffective. To address this, we propose a unified algorithmic framework featuring two algorithms: MIN-UCB-VI for regret minimization and MAX-LCB-VI for best policy identification. Both algorithms leverage fine-grained bias information to more effectively exploit offline data under general transition shifts. We provide theoretical guarantees for our framework, including both instance-dependent and independent upper bounds on regret and sub-optimality gap. Furthermore, we establish matching lower bounds to demonstrate the optimality of our approach and validate our theoretical findings through extensive experiments.
