Search papers, labs, and topics across Lattice.
This paper addresses the challenge of sample efficiency in image-based reinforcement learning by introducing Novelty and Surprise Prioritized Experience Replay (NSPER), which prioritizes experiences based on novelty and surprise to enhance exploration and learning. The authors also propose NSPER+R, which incorporates these signals as intrinsic rewards, leading to improved replay quality. Experimental results on DeepMind Control Suite tasks demonstrate that both NSPER and NSPER+R significantly enhance training efficiency and convergence speed compared to traditional methods.
Integrating novelty and surprise in experience replay can dramatically accelerate learning in image-based reinforcement learning environments.
Sample efficiency is a central challenge in reinforcement learning (RL), particularly in image-based domains where agents must learn from high-dimensional visual inputs. Traditional sampling often relies on random or suboptimal experience selection, leading to redundant updates and slow learning. Improving efficiency requires mechanisms that prioritize informative experiences while also encouraging effective exploration. Prioritized Experience Replay (PER) addresses part of this challenge by reusing high-value transitions, while intrinsic rewards promote the exploration of novel or uncertain states. However, their integration has not been extensively studied. This paper introduces Novelty and Surprise Prioritized Experience Replay (NSPER), which uses novelty to capture underrepresented states and surprise to expose gaps in the agent's understanding of the environment. We further extend this with NSPER+R, integrating these signals as intrinsic rewards to jointly improve replay quality and exploration. Experiments on DeepMind Control Suite tasks show that NSPER and NSPER+R improve training efficiency and convergence speed compared to existing methods in image-based RL.
