Search papers, labs, and topics across Lattice.
This paper introduces Aggregate-then-Calibrate (AtC), a two-stage framework designed to enhance human-centered assessment tasks by effectively combining human judgments and model-generated scores. The first stage aggregates diverse comparative judgments into a consensus ranking, while the second stage calibrates predictive model scores to maintain ordinal consistency and maximize quantitative information retention. Theoretical guarantees demonstrate that AtC outperforms traditional methods, achieving higher accuracy and robustness in scenarios where ground truth is limited or unavailable.
By integrating human judgment with model predictions, AtC achieves superior assessment accuracy, even in the absence of verifiable ground truth.
Human-centered assessment tasks, which are essential for systematic decision-making, rely heavily on human judgment and typically lack verifiable ground truth. Existing approaches face a dilemma: methods using only human judgments suffer from heterogeneous expertise and inconsistent rating scales, while methods using only model-generated scores must learn from imperfect proxies or incomplete features. We propose Aggregate-then-Calibrate (AtC), a two-stage framework that combines these complementary sources. Stage-1 aggregates heterogeneous comparative judgments into a consensus ranking using a rank-aggregation model that accounts for annotator reliability. Stage-2 calibrates any predictive model's scores by an isotonic projection onto the order, enforcing ordinal consistency while preserving as much of the model's quantitative information as possible. Theoretically, we show: (1) modeling annotator heterogeneity yields strictly more efficient consensus estimation than homogeneity; (2) isotonic calibration enjoys risk bounds even when the consensus ranking is misspecified; and (3) AtC asymptotically outperforms model-only assessment. Across semi-synthetic and real-world datasets, AtC consistently improves accuracy and robustness over human-only or model-only assessments. Our results bridge judgment aggregation with model-free calibration, providing a principled recipe for human-centered assessment when ground truth is costly, scarce, or unverifiable.
