Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
We introduce a formal criterion for exploration based on counterfactual information acquisition, demonstrating that common intrinsic reward objectives (count-based, prediction error, empowerment, information gain) can lead to Pareto-suboptimal exploration in a simple environment. The paper explains the mechanisms behind these failures and derives conditions for optimal exploration, also proposing a novel intrinsic reward objective that directly optimizes for improvements under the new criterion. This work highlights a critical disconnect between maximizing current intrinsic rewards and acquiring the most informative experiences for future learning.
Standard intrinsic rewards can fail to drive optimal exploration, even in simple settings, by neglecting the value of counterfactual information.
Intrinsic rewards are designed to guide exploration in reinforcement learning by assigning value to an agent's experience, for example through prediction error or learning progress. However, maximizing these rewards need not produce the most informative experience available. We propose a formal criterion for exploration that compares policies by the counterfactual information they acquire: how well their histories can substitute for experience under alternative policies. We construct a single, simple environment in which specified count-based, prediction-error, empowerment, and information-gain objectives have maximizing policies that are Pareto-suboptimal at acquiring counterfactual information. We explain these failures and establish conditions under which existing intrinsic rewards successfully encourage optimal exploration. We also construct an objective that assigns a higher value whenever exploration strictly improves under our criterion.
Stanford HAI