Search papers, labs, and topics across Lattice.
This paper introduces the LLM Nominal Response Model (LLM-NRM), a novel psychometric framework that evaluates Large Language Models (LLMs) by analyzing their preferences among incorrect multiple-choice answers, rather than solely focusing on correct responses. By leveraging a comprehensive dataset of 31,554 items across 14 benchmarks, LLM-NRM significantly outperforms traditional binary scoring methods, achieving a Spearman correlation of 0.920 with human-preference rankings. The findings reveal that incorrect answers provide valuable insights into model behavior, allowing for more nuanced ability estimates and efficient benchmarking with a drastic reduction in item count while maintaining performance integrity.
Incorrect answers reveal critical insights about LLM behavior, enabling a more refined understanding of model capabilities that binary scoring overlooks.
Most multiple-choice question (MCQ) benchmarks evaluate Large Language Models (LLMs) only by whether they select the correct answers. This binary scoring treats all incorrect responses alike, even though an LLM's preferences among incorrect options may contain systematic and useful information about its behavior and ability. We introduce the LLM Nominal Response Model (LLM-NRM), an option-aware psychometric framework that models the full distribution over answer choices to jointly estimate LLM ability and option-level item characteristics, while separating model-specific response calibration sharpness, positional preference, and difficulty-dependent fallback behavior. Across 189 LLMs and 31,554 items from 14 benchmarks, LLM-NRM predicts held-out LLM-item interactions more accurately than binary Item Response models and conventional nominal-response baselines, and its ability estimates achieve the strongest Spearman correlation of 0.920 with the external human-preference Arena.ai Elo leaderboard. Distractor identity contributes +101% additional Fisher Information per item beyond correctness, and incorrect responses alone recover full-information ability estimates with Spearman 0.943. The learned item parameters also enable efficient benchmarking, where 41 selected items preserve the full-bank ranking with Kendall's correlation 0.85, corresponding to a 770 times reduction. In conclusion, we show that incorrect answers carry distinct and useful measurement information rather than representing equivalent mistakes.
