Search papers, labs, and topics across Lattice.
This paper introduces PAWS, a novel segment-based preference learning method that addresses the distribution shift in preference-based reinforcement learning (PbRL) by aligning utility training with policy optimization. By utilizing segment-level advantage functions for policy updates, PAWS effectively preserves trajectory-level preference information and mitigates the reliance on unreliable per-step utility estimates. Experimental results in simulated robotic manipulation and locomotion tasks show that PAWS significantly outperforms existing PbRL methods, underscoring its effectiveness in improving policy learning through better temporal credit assignment.
Aligning utility training with policy optimization, PAWS overcomes distribution shifts that cripple traditional preference-based reinforcement learning methods.
Preference-based reinforcement learning (PbRL) learns policies from human trajectory-level comparisons, avoiding explicit reward design and expert demonstrations. Existing methods typically train utility functions on trajectory or segment-level preferences while relying on per-step utility estimates during policy optimization. This training and inference mismatch induces a distribution shift that severely degrades temporal credit assignment and limits policy learning. We analyze this issue and propose PAWS, a segment-based preference learning method that performs policy updates directly using segment-level advantage functions. By aligning utility training with policy optimization, PAWS preserves trajectory-level preference information and avoids unreliable per-step learning signals. Experiments on simulated robotic manipulation and locomotion tasks demonstrate that PAWS consistently outperforms existing PbRL approaches, highlighting the importance of distribution-consistent preference learning.
