Search papers, labs, and topics across Lattice.
This paper analyzes offline-to-online reinforcement learning with function approximation, proving a minimax lower bound that highlights the inherent difficulty of adaptation even with near-optimal offline Q-functions. To address this, they introduce a structural condition on offline value functions and propose O2O-LSVI, an algorithm that achieves improved sample complexity compared to pure online RL under this condition. Empirical results with neural networks validate the practical benefits of their approach.
Even with a near-optimal offline Q-function, adapting to a new environment can be as hard as learning from scratch, unless your offline data satisfies a new structural condition that this paper identifies.
We study value adaptation in offline-to-online reinforcement learning under general function approximation. Starting from an imperfect offline pretrained $Q$-function, the learner aims to adapt it to the target environment using only a limited amount of online interaction. We first characterize the difficulty of this setting by establishing a minimax lower bound, showing that even when the pretrained $Q$-function is close to optimal $Q^\star$, online adaptation can be no more efficient than pure online RL on certain hard instances. On the positive side, under a novel structural condition on the offline-pretrained value functions, we propose O2O-LSVI, an adaptation algorithm with problem-dependent sample complexity that provably improves over pure online RL. Finally, we complement our theory with neural-network experiments that demonstrate the practical effectiveness of the proposed method.
