Search papers, labs, and topics across Lattice.
This paper introduces an exposure-based reinforcement learning (RL) approach for learning-to-rank (LTR) that avoids the complexities of custom gradient computations, which are often a barrier for practitioners. By focusing on variance reduction and leveraging GPU capabilities, the authors achieve high sample efficiency through baseline corrections and partial marginalization, allowing for seamless integration with auto-differentiation. Experimental results demonstrate that this new method converges faster and yields significantly better ranking performance compared to traditional custom gradient methods, all without incurring additional computational costs.
High sample efficiency in learning-to-rank can be achieved without the headaches of custom gradients, making RL more accessible for practitioners.
Reinforcement learning (RL) methods for learning-to-rank (LTR) can optimize (almost) any ranking goal, e.g., from precision or discounted cumulative gain to fairness-of-exposure or ranking distillation. However, standard RL is ineffective and computationally costly due to the enormous action space in LTR settings. Existing methods reach computational efficiency through custom gradient computation algorithms, but they are very complex to implement and often clash with auto-differentiation. Consequently, existing RL for LTR is not attractive to many practitioners. We reconsider RL for LTR while actively avoiding reliance on custom gradients. Contrary to the existing approaches, we focus on variance reduction and GPU computation. In doing so, we discover that high sample-efficiency can be reached through baseline corrections and partial marginalization. Furthermore, we propose an abstraction that places gradient estimation behind a document-exposure distribution, this enables seamless plug-and-play integration with auto-differentiation. Thereby, one only has to implement a loss as a differentiable function of exposure and RL for LTR can optimize it using auto-differentiation. Our experimental results reveal that our new exposure-based RL for LTR approach converges considerably faster and at significantly higher ranking performance than existing custom gradients, with no additional costs in computation time when using GPUs. In contrast, existing custom gradients result in severe stability issues when converging over many epochs, which never occur for our methods. Thus, we considerably improve RL for LTR methodology by increasing its effectiveness, efficiency, and ease of application.
DeepMind