Search papers, labs, and topics across Lattice.
This paper introduces DualEval, a novel framework that jointly calibrates model abilities and item difficulties in a shared latent space, addressing the disconnect between static benchmarks and preference-based evaluations in LLM assessment. By applying DualEval across diverse domains and evaluating 18 advanced LLMs, the authors demonstrate that their approach yields reliable model rankings and insightful item-level diagnostics. The findings highlight the potential for more efficient evaluation processes, enabling applications such as benchmark compression and anomaly detection in model assessments.
DualEval reveals that unifying static and preference-based evaluations can lead to more reliable model rankings and deeper insights into item performance.
Current LLM evaluation relies on two complementary but often disconnected signals: static benchmarks with objective correctness labels and arena-style preference data that better reflect open-ended user interactions. We introduce DualEval, a latent model-item calibration framework that represents models and evaluation items in a shared space, jointly estimating model ability together with item difficulty and sharpness. We apply DualEval across four domains: coding, math, miscellaneous domain-knowledge tasks, and generic everyday user queries. Our evaluation uses 18 frontier LLMs, static benchmark labels, and reward-model scores validated against held-out human preferences for open-ended model responses. Empirically, our framework produces reliable and balanced model rankings, and its learned item-level profiles support downstream applications such as benchmark compression for sample-efficient evaluation and anomaly detection for contamination or outlier analysis. Overall, DualEval unifies static and arena-style evaluation through joint model-item calibration, producing model rankings and item-level diagnostics that support more sample-efficient, interpretable, and auditable evaluation pipelines.
