Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The authors conduct a psychometric audit of MMLU by applying Item Response Theory across 14,042 test items and 1,000 open-weights models to calibrate item difficulty against deterministic text-complexity metrics. Statistical testing via joint Wald tests reveals that MMLU conflates separable constructs—factual retrieval and reasoning depth—with difficulty mapping failing invariance between STEM and non-STEM partitions. Consequently, aggregate leaderboards track retrieval capacity rather than reasoning stability, causing Top-50 selections on aggregate score to displace roughly 22% of superior STEM-performing models.
Selecting top models via aggregate MMLU displaces roughly 22% of the best reasoning engines because the benchmark's single score fundamentally rewards factual recall over multi-step problem solving.
Although MMLU is widely adopted as a benchmark for calibrating general AI capabilities, we psychometrically demonstrate that its aggregate score primarily evaluates a model's factual retrieval capacity rather than its reasoning ability. By calibrating item difficulty for 1,000 open-weights language models over 14,042 MMLU test items using Item Response Theory, we show that evaluating both abilities via a single test is inherently flawed. Difficulty is then regressed on a deterministic, text-extractable framework of structural complexity. Applying a joint Wald test with subject-clustered covariances demonstrates that the MMLU conflates fundamentally separable constructs. The mapping from structural complexity to difficulty is not invariant across the benchmark's STEM and non-STEM partitions. This finding has practical consequences. Aggregate leaderboard ranks track non-STEM accuracy more closely than STEM accuracy, so selecting a Top-50 model on the aggregate for a reasoning-intensive deployment displaces roughly 22% of the STEM-appropriate choices. Furthermore, when controlling for the multiple-choice guessing floor natively inside the response model, we find that higher-ability models continue to degrade more steeply under increased reasoning depth. The MMLU aggregate therefore weights retrieval capacity and reasoning stability unequally, inadvertently favoring models optimized for retrieval. We release our deterministic framework as a reproducible auditing instrument and recommend disaggregated reporting.