Search papers, labs, and topics across Lattice.
This paper introduces SRPO, a Structure-aware Relative Policy Optimization framework that enhances reinforcement learning for listwise ranking by explicitly modeling the structural relationships among permutations. By employing a top-weighted Kendall-tau distance to assess discrepancies between sampled rankings, SRPO normalizes reward differences to focus on efficient local refinements, particularly in top-ranked positions. Experimental results show that SRPO significantly improves the effectiveness and stability of ranking systems, especially in scenarios with limited feedback and complex optimization requirements.
Ranking systems can achieve greater accuracy and stability by leveraging structural relationships in permutations, as demonstrated by the SRPO framework.
Ranking is a fundamental component of modern information access systems. Reinforcement learning (RL) provides a flexible framework for directly optimizing coarse-grained feedback and system-level objectives defined over the complete ranking list. However, existing RL-based ranking methods typically treat each sampled permutation as an atomic output and evaluate it primarily through a scalar reward, overlooking the structural relationships among different ranking lists. Consequently, permutations with similar rewards but substantially different permutation patterns may receive comparable optimization signals, potentially leading to inaccurate credit assignment and overly aggressive policy updates. To address this limitation, we propose SRPO, a \textbf{S}tructure-aware \textbf{R}elative \textbf{P}olicy \textbf{O}ptimization framework for listwise ranking. SRPO measures the discrepancy between sampled permutations using a top-weighted Kendall-tau distance and normalizes their pairwise reward differences by the corresponding distances. It quantifies the reward improvement per unit of ranking change, thereby emphasizing efficient local refinements, particularly those involving top-ranked positions. Experimental results across two ranking scenarios demonstrate that explicitly modeling permutation-level differences improves the effectiveness and stability of listwise ranking, with particularly favorable performance in limited-feedback and complex list-level optimization settings.
Tsinghua AI