Search papers, labs, and topics across Lattice.
This paper investigates the impact of training order on the decision-making capabilities of LLM scorers, revealing that equal ranking quality does not guarantee consistent decisions across different orderings. Through the introduction of Order-consistency SFT (OC-SFT), the authors demonstrate that it effectively mitigates order dependence while maintaining ranking quality, leading to improved decision stability across multiple tasks. The results show that OC-SFT significantly reduces the variability in retained sets compared to traditional methods, underscoring the importance of considering decision outcomes alongside ranking metrics.
Equal ranking quality can lead to drastically different decision outcomes, with OC-SFT providing a solution that enhances stability in LLM scoring.
Rerankers, reward models and multi-document QA scorers score candidate documents or responses in one LLM prompt, so each score depends on their order. Such scorers are selected on ranking quality, but their scores determine a decision: what a score threshold retains, a reader answers, or a preference model selects. However, equal ranking quality does not imply equal decisions: on passage reranking, five trained scorers within 0.010 nDCG@10 retain sets that overlap by only 0.66-0.84 when reordered. A published reranker takes the highest retained-set F1 in our comparison and still overlaps by only 0.667. No prompt-time change we test removes that order dependence: the only one that gains ranking quality leaves all three decisions unchanged. Order-consistency SFT (OC-SFT) attenuates it in the weights, training a candidate's score not to depend on the order. It holds ranking quality and leads every decision-stability measure among trained scorers on all three tasks: it flips the reader's answer on 0.125 of permutation pairs against 0.149-0.164 for three other objectives that target order. It is more stable than order-averaged distillation on 12 base models, and one OC-SFT permutation retains sets that overlap more than ten averaged off-the-shelf permutations. A comparison should therefore report what a threshold retains and a reader answers, not ranking quality alone. Code is available at https://github.com/thomsonreuters/presentation-dependence.
