Search papers, labs, and topics across Lattice.
This paper critiques the reliability of multiple-choice question answering (MCQA) benchmarks for evaluating large language models, highlighting their sensitivity to answer phrasing which can misrepresent a model's true knowledge. By employing a controlled testbed of models ranging from 1B to 8B parameters, the authors reveal that standard scoring methods can inaccurately indicate performance gaps of over 2 points, despite identical knowledge across models. To address this issue, they introduce ParaEval, an evaluation framework that utilizes multiple paraphrases for each answer option, successfully reducing the false performance gap to below 1 point, even in larger models like 70B and 120B.
Standard MCQA metrics can mislead evaluations by over 2 points due to phrasing sensitivity, but ParaEval cuts this gap to under 1 point, revealing true model capabilities.
Multiple-choice (MCQA) benchmarks are the standard for evaluating pretrained large language models, but their reliance on log-likelihood scoring makes them unreliable. Specifically, standard scores are highly sensitive to the exact phrasing (surface form) of the answers, conflating a model's familiarity with a specific phrase with its actual capability. We demonstrate this flaw using a controlled testbed of 1B-8B models trained on the same knowledge. Despite having identical knowledge, standard metrics falsely report a performance gap of over 2 points. To solve this, we propose ParaEval, an evaluation framework that queries models using multiple paraphrases per answer option. By scoring each model based on its most favorable phrasing, ParaEval successfully reduces the false performance gap to below 1 point. We confirm that these evaluation artifacts, and the improvements from ParaEval, persist in frontier 70B and 120B open-source models. Ultimately, ParaEval provides a robust and efficient way to evaluate true underlying capability rather than surface-form familiarity.
Meta AI