Search papers, labs, and topics across Lattice.
This study investigates the impact of label-free strategies on the accuracy of large language models in multiple-choice questions, revealing that traditional MCQ scores are influenced by option order rather than pure knowledge. By employing two distinct strategies鈥攇eneration-then-matching and isolated scoring鈥攖he authors find that neither approach consistently enhances accuracy, indicating that the challenge lies in the withholding of options rather than the matching process itself. Notably, while showing all options with an LLM matcher maintains baseline performance, eliminating positional bias does not guarantee accuracy improvements, although cyclic permutation can lead to better outcomes.
Label-free strategies fail to improve accuracy in MCQ evaluations, revealing that option withholding is a critical bottleneck in assessing model knowledge.
Multiple-choice benchmarks are widely used to evaluate large language models, but MCQ scores conflate knowledge with sensitivity to option order, which makes them unreliable measures of model knowledge. In this paper, we test whether preventing a model from seeing option labels while committing to an answer removes positional influence and, in turn, improves performance. We evaluate two different strategies for mitigating bias. The first uses a generation-then-matching approach, and the second scores options in isolation, which is positionally unbiased by construction. Neither reliably improves accuracy. A complete decomposition shows that the bottleneck is withholding options, not the matching step. The only configuration that consistently matches the baseline is the one that shows the model all options paired with an LLM matcher. However, eliminating positional influence entirely still does not reliably yield accuracy gains, while cyclic permutation often improves them. For two-stage prompting, an aggregate measure of recall imbalance and a direct per-question measure of order sensitivity both fail to show reliable debiasing.
