Search papers, labs, and topics across Lattice.
This paper addresses the mismatch between dual-encoder retrievers optimized for contrastive similarity and downstream rerankers focused on fine-grained relevance in e-commerce retrieval. The authors introduce PAO (Positive-Advantage-Only), a selective reinforcement learning method that mitigates the degradation of embedding geometry caused by standard policy-gradient updates in a frozen document index. Experiments show that PAO not only preserves the global topological stability of embeddings but also significantly enhances retrieval performance compared to standard RL and distillation baselines.
Standard reinforcement learning can degrade embedding quality in retrieval tasks, but PAO selectively optimizes only the most relevant items, leading to superior performance.
In large-scale e-commerce retrieval, dual-encoder retrievers are op- timized for contrastive similarity, whereas downstream rerankers capture finer-grained relevance preferences; this objective mis- match limits end-to-end retrieval quality. Reinforcement Learning offers a way to use reward-model feedback for retriever adaptation, but we observe that standard policy-gradient updates can degrade embedding geometry, especially when the document index must remain frozen due to industrial constraints. To address this, we propose PAO (Positive-Advantage-Only), a selective RL optimization method. Our analysis reveals that in- discriminate penalization of negative samples (pushing away) in a frozen high-dimensional space disrupts pre-trained semantic man- ifolds. PAO selectively applies gradient updates only to retrieved items with positive advantages, effectively pulling query embed- dings toward high-reward regions while preserving global topo- logical stability. Experiments on both a massive industrial dataset and public benchmarks demonstrate that PAO significantly outper- forms standard RL and distillation baselines.
