Search papers, labs, and topics across Lattice.
This study investigates the impact of pruning on large language models (LLMs) by comparing their performance on multiple-choice evaluations versus open generation tasks. The authors reveal a "benchmark illusion," where pruned models can select correct answers in multiple-choice settings but struggle to generate those answers in open-ended formats due to demotion rather than complete erasure. The findings highlight that traditional benchmarks may misrepresent the usability of compressed models, emphasizing the need for evaluations that assess generative capabilities alongside recognition tasks.
Pruned LLMs can ace multiple-choice tests while failing to generate correct answers, revealing a critical evaluation blind spot in model assessment.
Compressing large language models reduces memory use and inference cost, but it can also create failures that standard benchmarks miss. A pruned model may still perform well on multiple-choice evaluations, yet fail to answer the same question in open generation. We ask what pruning changes: does it erase the correct answer, or does it make the answer harder to produce as the top output? We study this question with multilingual question answering, tracking the same questions before and after pruning. We find a benchmark illusion. Under high-sparsity pruning, especially Wanda, models often fail in greedy open generation while still selecting the correct answer under multiple-choice scoring. In these recognition-only errors, the answer is usually not gone, but demoted: it often reappears with beam search, sampling, or one in-context example. Overall, multiple-choice benchmarks can overstate the usability of compressed LLMs, creating an evaluation blind spot. Compressed models should be tested on what they can produce, not only on what they can recognize.
