Search papers, labs, and topics across Lattice.
This paper investigates the necessity of pretraining Q-functions in the context of online reinforcement learning (RL) fine-tuning, challenging the conventional wisdom that suggests it is beneficial. The authors find that using a randomly-initialized Q-function can yield highly effective policies, as naive pretraining often fails to align with the Q-function that online fine-tuning ultimately converges to. They introduce Initialization via Policy Ensemble (IPE), which enhances performance by leveraging diverse policies to improve Q-function learning, achieving an average 1.26x performance boost over traditional pretraining methods across various continuous control tasks.
Naive pretraining of Q-functions may hinder performance, while a simple ensemble approach can lead to over 1.26x improvement in fine-tuning effectiveness.
Pre-training followed by fine-tuning has become the dominant recipe for learning performant policies, and in value-based reinforcement learning (RL) this raises a natural question: given a pretrained policy, should the Q-function be pretrained on offline data too? Conventional wisdom suggests it should, but recent results show that online RL with a randomly-initialized Q-function can result in highly performant and reliable policies without needing to pretrain the Q-function. In this paper, we systematically study whether pretraining the Q-function actually helps when fine-tuning on top of a pretrained base policy. We find, surprisingly, that naive Q-function pretraining often provides little benefit over random initialization. We show this stems from a fundamental mismatch: the Q-function learned during pretraining targets the pretrained policy's Q-function, not the Q-function that online fine-tuning converges to, and this gap persists even after offline value maximization. Motivated by this finding, we propose Initialization via Policy Ensemble (IPE), a simple method that trains multiple diverse policies and uses their pooled rollouts to bootstrap the Q-function learning in online RL. Across a suite of challenging continuous control benchmarks, IPE yields an average 1.26x improvement in fine-tuning performance over naive Q-function pre-training.
Stanford HAI