Search papers, labs, and topics across Lattice.
This paper introduces Critic-Free Pretraining (CFP), a novel approach for online reinforcement learning that eliminates the reliance on offline-trained critics, which can misalign with rapidly changing online environments. By initializing a new critic that adapts solely based on online interactions, the method enhances policy improvement and exploration efficiency. The results demonstrate that CFP not only matches but often surpasses traditional offline-to-online algorithms across a variety of challenging tasks, indicating significant potential for real-world applications.
Abandoning biased offline critics leads to more efficient online reinforcement learning, achieving superior performance on challenging tasks.
Offline-to-online (O2O) reinforcement learning aims to leverage policies pretrained on static datasets while improving them through online interaction. However, directly reusing an offline-trained critic can hinder online fine-tuning: as the policy and data distribution change rapidly, value estimates inherited from offline training may become misaligned with the online environment, leading to inaccurate policy improvement and inefficient exploration. To address this problem, we introduce \textbf{C}ritic-\textbf{F}ree \textbf{P}retraining: an efficient paradigm that completely abandons the approach of offline critic training, allowing a freshly initialized critic to adapt without inheriting biased estimates. CFP is compatible with various mainstream O2O algorithms and consistently matches or improves upon conventional O2O algorithms across a diverse set of tasks, with particularly pronounced gains on several challenging tasks.
