Search papers, labs, and topics across Lattice.
This paper introduces multi-expert Conformal Risk Control (CRC) algorithms for evaluating open-ended dialogue using pairwise LLM judging, focusing on the aggregation of multiple expert assessments. By employing Score Averaging and Decision Voting methods, the authors demonstrate that these strategies outperform single-expert approaches on homogeneous panels, while also addressing the limitations of uniform thresholds in heterogeneous expert settings. The proposed Marginal-Calibrated Conformal Consensus (MC3) method effectively captures distinct scoring scales from different experts, significantly enhancing accuracy and acceptance rates across diverse datasets.
Multi-expert aggregation not only mitigates risk at the decision threshold but also optimizes scoring functions, leading to substantial improvements in dialogue evaluation accuracy.
In this paper, we explore multi-expert Conformal Risk Control (CRC) algorithms for pairwise LLM-as-a-Judge evaluation in open-ended dialogue. Our core insight is that multi-expert aggregation offers a complementary remedy to CRC: whereas CRC controls risk at the decision threshold through abstention, aggregation sanitizes the scoring function at its source. Guided by this, we first design two multi-expert CRC methods: Score Averaging and Decision Voting, which aggregate at the score and decision levels, respectively. While both strategies outperform single-expert methods on homogeneous expert panels, on heterogeneous LLM judges they remain risk-valid but recover only limited coverage, because a uniform threshold cannot match the experts'distinct scoring scales. To resolve this issue, we further propose Marginal-Calibrated Conformal Consensus (MC3): it captures distinct per-expert scales via initial threshold ratios, while jointly tuning a unified decision function $C_t(x)$ applied identically in both calibration and test, thereby preserving exchangeability. To evaluate our framework, we construct Panel, a 1,800-pair human pairwise-preference benchmark for open-ended dialogue. It is built on responses generated by four open-weight LLMs over dialogue contexts from three domains (ESConv, MSC, DREAM), with full logit access. In experiments, we find that both Score Averaging and Decision Voting substantially improve accuracy and acceptance rate on homogeneous panels. Notably, MC3 extends these gains to heterogeneous panels by accommodating distinct per-expert scoring scales across all three datasets.
