Search papers, labs, and topics across Lattice.
This paper introduces QDHUAC, a novel target-free distributional Quality-Diversity Reinforcement Learning algorithm that enhances sample efficiency and accelerates training in complex locomotion tasks. By eliminating the need for target networks, QDHUAC achieves high Update-to-Data ratios, allowing for stable training with significantly fewer environment steps compared to traditional methods. The results demonstrate that QDHUAC not only maintains competitive performance in coverage and fitness on high-dimensional Brax environments but also sets a new benchmark for sample efficiency in evolutionary RL algorithms.
Achieving robust Quality-Diversity in RL without the computational burden of target networks could revolutionize how we approach skill discovery in complex environments.
Quality-Diversity (QD) algorithms excel at discovering diverse repertoires of skills, but are hindered by poor sample efficiency and often require tens of millions of environment steps to solve complex locomotion tasks. Recent advances in Reinforcement Learning (RL) have shown that high Update-to-Data (UTD) ratios accelerate Actor-Critic learning. While effective, standard high-UTD algorithms typically utilise target networks to stabilise training. This requirement introduces a significant computational bottleneck, rendering them impractical for resource-intensive Quality-Diversity (QD) tasks where sample efficiency and rapid population adaptation are critical. In this paper, we introduce QDHUAC, a sample-efficient, target-free and distributional QD-RL algorithm that provides dense and low-variance gradient signals, which enables high-UTD training for Dominated Novelty Search whilst requiring an order of magnitude fewer environment steps. We demonstrate that our method enables stable training at high UTD ratios, achieving competitive coverage and fitness on high-dimensional Brax environments with an order of magnitude fewer samples than baselines. Our results suggest that combining target-free distributional critics with dominance-based selection is a key enabler for the next generation of sample-efficient evolutionary RL algorithms.
