Search papers, labs, and topics across Lattice.
This paper introduces a novel framework that integrates vision-language model (VLM) generated preferences with the Plackett-Luce (PL) model for preference-based reward learning in reinforcement learning. By leveraging listwise rankings instead of the traditional pairwise comparisons, the approach allows for a more comprehensive utilization of VLM capabilities, leading to improved training of robotic policies. Evaluations on Meta-World manipulation tasks reveal that the PL reward models consistently match or exceed the performance of existing pairwise methods, achieving an impressive 86% mean final success rate in certain scenarios.
Listwise preference-based reward learning from vision-language models can outperform traditional pairwise methods, achieving up to 86% success in complex robotic tasks.
Vision-language models (VLMs) have emerged as a powerful source of supervision for reinforcement learning, enabling agents to leverage rich semantic knowledge during training. Inspired by the success of preference-based reward learning (PbRL) in reinforcement learning from human feedback (RLHF), vision-language model generated image-based preferences provide an effective source for learning reward functions. This can be done by visually comparing two outcomes through the Bradley-Terry (BT) model. However, this pairwise formulation utilizes only two observations at a time, despite VLMs being capable of ranking multiple candidates. The Plackett-Luce (PL) formulation can shape a reward model with listwise rankings as opposed to pairwise preferences, allowing for a more suited use of a VLM based ranking. In this work, to our knowledge, we introduce the first framework that combines VLM-generated preferences with the Plackett-Luce model for reward learning. We evaluate our approach on Meta-World manipulation tasks and show that Plackett-Luce (PL) reward models can train robotic policies from VLM-generated rankings as effectively as pairwise Bradley-Terry, $K$-wise Bradley-Terry, and RL-VLM-F baselines. Across all environments, at least one PL ranking size ($K \in \{3,4,5\}$) consistently performs with or outperforms other methods in mean success rate. Unlike pairwise methods, which are restricted to $K=2$, PL supports different ranking sizes and can therefore be adapted to the environment and desired feedback format. Our best PL configuration achieves an 86% mean final success rate and matches the Oracle baseline on Drawer Open. Overall, these results demonstrate that listwise VLM preference supervision is a competitive and flexible approach to reward learning for reinforcement learning.
