Search papers, labs, and topics across Lattice.
This paper explores the limitations of the LLM-as-a-judge (LLMaJ) framework in evaluating human judgment distributions (HJD) by highlighting its poor performance in predicting soft-labels despite achieving near-human accuracy on hard-labels. The authors introduce NAPHA (eNtropy-Aware Post-Hoc Alignment), a novel method that aligns LLM outputs with HJD by categorizing instances into discrete entropy classes and using specialized models for alignment. Results demonstrate that NAPHA significantly enhances soft-label prediction, particularly for high-entropy instances, thereby improving the model's ability to capture diverse human perspectives.
LLMs can achieve near-human performance on hard-labels but struggle with soft-labels, revealing a critical gap in automatic evaluation methods that NAPHA effectively bridges.
The LLM-as-a-judge (LLMaJ) framework offers a cost-effective and reproducible solution for automatic evaluation. However, current evaluation practices typically compare LLMaJ judgments against aggregated ground-truth labels, overlooking the valuable information contained in Human Label Variation (HLV). Inspired by an increasing line of work that proposes to leverage HLV, we systematically study LLMaJ performance on predicting both a single, aggregated ground truth hard-label and unaggregated soft-labels that represent Human Judgment Distributions (HJD). Our results across five diverse datasets reveal that while LLMs achieve near human-level performance at hard-label prediction on most tasks, they exhibit poor performance when predicting soft-labels. To address this limitation, we propose NAPHA (eNtropy-Aware Post-Hoc Alignment), a simple yet effective lightweight post-hoc alignment method that matches the LLM distribution to the HJD by first assigning an instance to a discrete entropy class and then routing it to specialized, trained alignment models. We find that NAPHA consistently improves soft-labels prediction across base LLM models and datasets, with particularly strong gains on high-entropy instances where capturing diverse human perspectives is most critical. We also show via oracle experiments that improving entropy class prediction can substantially enhance NAPHA's practical effectiveness.
