Search papers, labs, and topics across Lattice.
This paper introduces Uncertainty-Balanced Preference Planning (UBP2), a model-based approach that enhances sample efficiency in preference-based reinforcement learning by actively directing exploration through uncertainty reasoning in reward, dynamics, and value functions. By employing ensembles of models to evaluate trajectories based on a unified score that incorporates expected reward, terminal value, and epistemic uncertainty, UBP2 effectively balances exploitation and information acquisition. Experimental results on the Meta-World benchmark demonstrate that UBP2 significantly outperforms both model-free preference-based methods and traditional model-based approaches in terms of sample efficiency.
UBP2 achieves a remarkable boost in sample efficiency for preference-based RL by actively balancing exploration and exploitation through uncertainty reasoning.
Preference-based RL provides an approach to learning reward models from pairwise comparisons of behaviors, bypassing the need for explicit reward design. However, existing methods typically rely on passive data collection and suffer from poor sample efficiency, especially during the early stages of learning. We introduce a model-based approach that actively directs exploration by jointly reasoning over uncertainties in the reward, dynamics, and value functions. Our method, Uncertainty-Balanced Preference Planning (UBP2), uses ensembles of reward, dynamics, and value function models to evaluate candidate trajectories according to a unified score that combines expected reward, terminal value, and epistemic uncertainty. Planning under this objective yields an explicit tradeoff between exploitation and information acquisition without requiring ad hoc exploration heuristics. Under standard regularity assumptions, we establish sublinear regret guarantees for both finite-horizon and infinite-horizon settings. Empirically, experiments on the Meta-World benchmark show UBP2 achieves substantially higher sample efficiency than model-free preference-based methods and non-optimistic model-based baselines.
