Search papers, labs, and topics across Lattice.
This paper introduces PreferenceEKF, a novel method for efficiently learning reward models from human preferences in reinforcement learning by framing the problem as a sequential Bayesian filtering task. By utilizing an extended Kalman filter within a low-dimensional parameter subspace, the method significantly improves uncertainty quantification and sample efficiency, allowing for scalable sampling of neural network parameters. Experiments demonstrate that PreferenceEKF outperforms existing Bayesian deep learning methods in terms of sample efficiency, runtime, and calibration, leading to competitive performance in offline reinforcement learning tasks.
Achieving better sample efficiency and scalability in reward learning, PreferenceEKF redefines how we approach uncertainty in reinforcement learning from human feedback.
Reinforcement learning from human feedback (RLHF) has emerged as a powerful yet sample-inefficient approach for learning reward models from human preferences, making active learning a critical component in synthesizing informative preference queries. However, effective uncertainty quantification required for active learning remains a key challenge for large neural network reward models. In this paper, we introduce PreferenceEKF, a sample-efficient approach that tracks reward model uncertainty by framing active preference learning as a sequential Bayesian filtering problem. Instead of relying on computationally prohibitive posterior inference over the full neural network parameter space, our method performs sequential inference via an extended Kalman filter within a low-dimensional parameter subspace, continuously updating the reward model posterior as new preference queries arrive. Our approach enables scalable sampling of neural network parameters to efficiently compute acquisition functions for active reward learning. Experiments on the D4RL and V-D4RL benchmarks demonstrate that our approach achieves better sample efficiency, runtime, scalability, and calibration compared to other Bayesian deep learning approaches, and the learned reward models lead to competitive offline reinforcement learning policy performance. This highlights the potential of scalable Bayesian methods for preference-based reward modeling in RLHF. Our code is available at https://github.com/yutaizhou/bnn_pref.
