Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces the Expert Behavior Prior (EBP) algorithm, which enhances sample efficiency in online reinforcement learning by generating expert policy priors from an online replay buffer rather than relying on static offline datasets. By employing a Q-guided conditional variational autoencoder (Q-CVAE) and an expert policy guidance (EPG) mechanism, EBP facilitates the selection of high-value actions for policy updates, resulting in improved policy exploitation and stability during training. Experimental results on robotic and industrial control benchmarks show that EBP significantly outperforms existing online RL methods, achieving superior sample efficiency and convergence stability.
Generating expert policy priors directly from online data can revolutionize sample efficiency and stability in reinforcement learning.
Behavior prior reinforcement learning (BPRL) has emerged as a promising paradigm to improve sample efficiency in online reinforcement learning (RL) by leveraging policy priors derived from offline demonstrations. However, most existing BPRL methods rely on static offline datasets, which often suffer from low data diversity and suboptimal trajectory quality. This reliance restricts the effectiveness of policy priors, hindering both policy exploitation and stability during online training. Consequently, agents are prone to inefficient exploration and unstable learning dynamics. To address these limitations, we deviate from existing offline pre-training methods and propose an Expert Behavior Prior (EBP) algorithm. Specifically, we introduce a Q-guided conditional variational autoencoder (Q-CVAE) that learns to generate expert policy priors directly from the online replay buffer. This enables the generation of high-value actions for guiding policy updates without relying on pre-collected expert trajectories. To further enhance policy exploitation, we propose an expert policy guidance (EPG) mechanism that selects expert actions from a generative support set, and we integrate a policy gradient correction (PGC) module to harmonize Q-guidance with expert supervision, promoting stable and consistent policy improvement. Extensive experiments conducted on robotic control (Gym, PyBullet) and industrial control (DMControl) benchmarks demonstrate that EBP significantly outperforms state-of-the-art online RL algorithms, achieving higher sample efficiency and more stable convergence.