Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper expands the framework of preference-based reinforcement learning by introducing a setting where human experts can label trajectory pairs as incomparable, thereby addressing a common limitation in traditional RL approaches. The authors propose a Bradley-Terry-inspired rationality model that captures these incomparabilities and infers a multi-dimensional reward function, providing a comprehensive analysis of the model's properties and sample complexity. Experimental results demonstrate the model's effectiveness in reconstructing reward functions that align with expert comparisons and recovering the Pareto frontier of policies, showcasing its robustness across different levels of expert rationality.
Experts can now express trajectory comparisons as incomparable, fundamentally enhancing how we model human preferences in reinforcement learning.
In this work, we study the reinforcement learning (RL) problem from pairwise trajectory comparisons provided by a human expert. We generalize preference-based RL by formalizing a novel setting in which the expert can also label trajectory pairs as incomparable, i.e., when neither trajectory dominates the other. We introduce the learning problem and the desiderata that its solution should satisfy. Then, we propose a novel Bradley-Terry-inspired rationality model that effectively captures incomparabilities and infers a multi-dimensional reward function, and we study its properties. We provide a sample complexity analysis for learning the model parameters when a dataset is available. Finally, we evaluate our model's ability to reconstruct a reward function that aligns with the expert's comparisons in simulated environments and to recover the Pareto frontier of policies, along with a robustness analysis across varying levels of expert rationality.