Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
While epistemic value functions offer a principled foundation for optimistic exploration in reinforcement learning, their deep approximations consistently suffer from severe representation, temporal propagation, and optimization failures. To resolve these vulnerabilities, the authors formulate DEVOTE, a model-free algorithm that controls out-of-distribution uncertainty generalization, stabilizes temporal Bellman propagation, and adapts to non-stationary exploration targets. Across challenging reward-free and continuous-control benchmarks, DEVOTE reliably out-explores and outperforms leading model-free and model-based baselines.
Value-driven optimistic exploration has historically crumbled under deep network approximations, but diagnosing and repairing failure modes in uncertainty propagation allows pure model-free RL to beat complex model-based exploration baselines.
Principled exploration in reinforcement learning requires an agent to quantify its epistemic uncertainty and act to resolve it. Uncertainty over the value function provides a natural signal for exploration, yet existing deep approximations remain brittle and perform inconsistently. The central challenge is therefore to scale these ideas robustly. We conduct a systematic empirical study of how epistemic uncertainty is represented, propagated, and optimized in deep epistemic value functions, and uncover distinct failure modes along each of these axes. These findings motivate DEVOTE, a model-free reinforcement learning algorithm that controls how uncertainty generalizes beyond observed data, stabilizes its temporal propagation, and preserves adaptation to the resulting non-stationary exploration objective. Across reward-free exploration and challenging continuous-control tasks, DEVOTE reaches novel states more effectively and achieves higher task return than strong model-free and model-based exploration baselines. These results provide evidence that deep epistemic value functions are a promising path toward scalable, principled exploration.
ETH
Max Planck