Search papers, labs, and topics across Lattice.
This paper introduces a Localize-Then-Decide framework to improve the reliability of large language models (LLMs) in evaluating output quality and preference alignment. By first using conformal prediction to create a shortlist of candidate responses and then applying a calibrated confidence-based rule, the method restores the relationship between confidence and disagreement risk with human judgments. Experimental results show that this approach significantly enhances guarantee success rates and coverage compared to traditional single-stage methods, particularly as the number of candidate responses increases.
Confidence estimates from LLMs can be misleading when evaluating many candidates, but a new framework ensures high-probability agreement with human judgments.
Large language models (LLMs) are increasingly used as evaluators to assess output quality and preference alignment, yet providing reliable guarantees of agreement with human judgments remains challenging. Recent work introduces confidence-thresholding methods that provide such guarantees for pairwise comparisons, relying on the assumption that higher estimated confidence implies lower disagreement risk with humans. However, this assumption can break down when the number of candidate responses increases, since distributing probability mass across many alternatives can distort confidence estimates. To address this issue, we propose a Localize-Then-Decide framework. First, conformal prediction localizes a small shortlist that contains the human-preferred response with high probability. Then, a calibrated confidence-based rule selectively chooses a single response from this shortlist or abstains. This design restores the monotonic relationship between confidence and disagreement risk and enables high-probability agreement guarantees. Experiments with multiple candidate sizes across several datasets and judge LLMs demonstrate that our framework consistently achieves higher guarantee success rates and substantially higher coverage than single-stage baselines.
