Search papers, labs, and topics across Lattice.
This paper introduces a novel framework that integrates vision-language model (VLM) generated preferences with the Plackett-Luce (PL) model for preference-based reward learning in reinforcement learning. By leveraging listwise rankings instead of the traditional pairwise comparisons, the approach allows for a more comprehensive use of VLM capabilities, demonstrating that PL models can effectively train robotic policies across various tasks. The results indicate that at least one configuration of the PL model consistently matches or exceeds the performance of established pairwise methods, achieving an 86% mean final success rate in specific environments.
Listwise rankings from vision-language models can outperform traditional pairwise methods in training robotic policies, achieving competitive success rates with greater flexibility.
Vision-language models (VLMs) have emerged as a powerful source of supervision for reinforcement learning, enabling agents to leverage rich semantic knowledge during training. Inspired by the success of preference-based reward learning (PbRL) in reinforcement learning from human feedback (RLHF), vision-language model generated image-based preferences provide an effective source for learning reward functions. This can be done by visually comparing two outcomes through the Bradley-Terry (BT) model. However, this pairwise formulation utilizes only two observations at a time, despite VLMs being capable of ranking multiple candidates. The Plackett-Luce (PL) formulation can shape a reward model with listwise rankings as opposed to pairwise preferences, allowing for a more suited use of a VLM based ranking. In this work, to our knowledge, we introduce the first framework that combines VLM-generated preferences with the Plackett-Luce model for reward learning. We evaluate our approach on Meta-World manipulation tasks and show that Plackett-Luce (PL) reward models can train robotic policies from VLM-generated rankings as effectively as pairwise Bradley-Terry, $K$-wise Bradley-Terry, and RL-VLM-F baselines. Across all environments, at least one PL ranking size ($K \in \{3,4,5\}$) consistently performs with or outperforms other methods in mean success rate. Unlike pairwise methods, which are restricted to $K=2$, PL supports different ranking sizes and can therefore be adapted to the environment and desired feedback format. Our best PL configuration achieves an 86% mean final success rate and matches the Oracle baseline on Drawer Open. Overall, these results demonstrate that listwise VLM preference supervision is a competitive and flexible approach to reward learning for reinforcement learning.
