Search papers, labs, and topics across Lattice.
This paper introduces Video-based Optimal Transport Preference (VOTP), a semi-supervised framework that enhances preference-based reinforcement learning (PbRL) by utilizing optimal transport to align visual trajectories in the representation space of Video Foundation Models (ViFMs). By generating high-fidelity pseudo-labels from a limited number of human labels, VOTP significantly reduces the need for extensive human supervision while maintaining performance. Experimental results across locomotion and manipulation benchmarks reveal that VOTP outperforms existing offline PbRL methods, demonstrating its effectiveness even in challenging scenarios with visual distractors and real robotic tasks.
VOTP slashes the labeling burden in preference-based reinforcement learning, achieving superior performance with minimal human input.
Conveying complex objectives to reinforcement learning (RL) agents often requires meticulous reward engineering. Preference-based RL (PbRL) offers a promising alternative by learning reward functions from human feedback, but its scalability is hindered by high labeling costs. Inspired by advances in Video Foundation Models (ViFMs), we present Video-based Optimal Transport Preference (VOTP), a semi-supervised framework that learns effective reward functions from only a handful of labels. By leveraging optimal transport to align visual trajectories within the rich representation space of ViFMs, VOTP effectively generates high-fidelity pseudo-labels for large amounts of unlabeled data, substantially reducing human supervision. Extensive experiments across locomotion and manipulation benchmarks demonstrate the superiority of VOTP, which outperforms state-of-the-art offline PbRL methods under limited feedback budgets. We also showcase the robustness of VOTP in the presence of visual distractors and validate its utility on real robotic tasks, where it learns meaningful rewards with minimal human input.
