Search papers, labs, and topics across Lattice.
This paper addresses the systematic biases in LLM-as-a-Judge systems, particularly verbosity bias, by framing LLM evaluation as a positive-unlabeled learning problem. The authors introduce a geometric auditing framework based on Partial Optimal Transport to align human-verified positive judgments with a reliable subset of unlabelled outputs, enabling the identification of human-consistent preferences without the need for retraining. Experimental results show enhanced alignment with human preferences, improved robustness against presentation biases, and interpretable confidence estimates, highlighting a scalable solution for LLM evaluation.
LLM evaluation can be significantly improved by correcting biases without retraining, using a novel approach that leverages positive-unlabeled learning.
Large Language Models (LLMs) are increasingly used as judges for scalable evaluation, yet such LLM--as--a--Judge systems exhibit systematic biases that are decoupled from semantic quality, most notably verbosity bias. Meanwhile, human supervision is costly and typically selective, yielding reliable positive judgments but leaving most outputs unlabelled and potentially mixed in quality. We formulate LLM evaluation under selective human supervision as a positive--unlabelled learning problem and propose a geometric auditing framework based on Partial Optimal Transport. By aligning a small set of human--verified positives with a reliable subset of unlabelled outputs in a fixed embedding space, our method identifies human--consistent preferences and corrects biased judges without retraining. Experiments demonstrate improved alignment with human preferences, increased robustness to presentation biases, and interpretable confidence estimates, offering a scalable and statistically grounded alternative to existing LLM--as--a--judge pipelines.
