Search papers, labs, and topics across Lattice.
This study investigates the impact of position bias on the consistency of preference rankings generated by listwise LLM-based rerankers in recommendation systems. By introducing a novel evaluation framework that measures pairwise preference instability, global preference inconsistency, and listwise output consistency, the authors reveal that LLMs are sensitive to the order of input candidates, leading to unreliable rankings. The findings indicate that merely improving relevance or balancing exposure across positions does not guarantee stable or coherent preferences, challenging the assumption that LLMs can serve as robust rerankers.
Position bias in LLM-based rerankers can lead to significant inconsistencies in preference rankings, undermining their reliability in recommendation systems.
Large language models (LLMs) have emerged as promising listwise rerankers for recommender systems, but their reliability under equivalent candidate permutations remains unclear. Since recommendation candidates form an unordered set, a reranker should not depend on the arbitrary order used to serialize them. However, decoder-only LLM rerankers can allow input order to affect model scores, pairwise preferences, and rankings. We study how position bias affects the ranking process induced by LLM-based rerankers. Instead of measuring only changes in final ranked lists, we treat rankings produced under equivalent candidate permutations as observations of an induced preference system. We introduce an evaluation framework measuring pairwise preference instability, global preference inconsistency, and listwise output consistency. This framework characterizes candidate-order sensitivity at the pairwise, global, and output levels. Experiments across multiple LLMs, datasets, and list lengths show that these consistency measures are closely aligned, but can diverge from recommendation effectiveness and marginal position-exposure bias. Improving relevance or flattening exposure across positions does not necessarily restore stable pairwise preferences, globally coherent preference structures, or consistent ranked outputs. These results show that reducing marginal exposure skew is insufficient to establish ranking-function validity in LLM-based reranking. Code is available at https://github.com/ejbito/InvariRank .
