Search papers, labs, and topics across Lattice.
This paper introduces a quantile-based ensemble method for finite-horizon Markov Decision Processes (MDPs) that circumvents the need for complex count-based uncertainty estimates, which are often impractical in real-world applications. By leveraging a recent ensemble method from Multi-Armed Bandits, the authors establish a theoretical foundation for this approach, demonstrating that it achieves optimal variance-dependent regret bounds. The key result shows that their count-free method not only simplifies the exploration process in RL but also maintains theoretical rigor, enhancing the understanding of ensemble-based exploration strategies.
Achieving optimal regret bounds without the complexity of count-based uncertainty estimates could revolutionize exploration strategies in reinforcement learning.
Optimal Reinforcement Learning (RL) algorithms typically rely on carefully constructed count-based uncertainty estimates to drive exploration. Although theoretically sound, such estimates are hard to compute in practical settings and therefore offer limited insight for designing exploration heuristics. Meanwhile, ensembling has emerged as a practical approach, but remains without theoretical justification. Building on a recent ensemble-based method for Multi-Armed Bandits, we propose a quantile-based ensemble method for finite-horizon Markov Decision Processes (MDPs). Our simple count-free approach achieves optimal variance-dependent regret bounds, providing theoretical grounding for ensemble-based exploration in RL.
Google Research