Search papers, labs, and topics across Lattice.
This paper introduces Cognitive Diagnostic Profiling (CDP), a zero-shot framework that enhances the psychometric calibration of educational tests by prompting large language models (LLMs) to simulate diverse examinee profiles. By evaluating various LLM configurations against human responses, the study demonstrates significant improvements in alignment across ability distribution, mastery profiles, and item difficulty. The results indicate that CDP can effectively bridge the gap between LLM-simulated and human examinees, achieving high correlation scores and reduced error metrics, particularly with reasoning-enabled models like Gemini 3.0 Flash.
LLMs can now simulate diverse cognitive profiles that align closely with human examinees, enhancing the reliability of educational test calibration.
Psychometric calibration for educational tests typically requires costly human response data. Large language models (LLMs) simulated examinees offer a promising route to early calibration, but their responses are too accurate and too uniform. We propose Cognitive Diagnostic Profiling (CDP), a zero-shot framework that prompts LLMs to simulate plausible examinees with diverse cognitive profiles: binary attribute-mastery patterns are rendered as natural-language profiles and sampled under an uninformative or an informative distribution. Using the Tatsuoka fraction-subtraction dataset (536 examinees, 15 items, five attributes), we evaluated eight LLM configurations under no-profile, uninformative-CDP, and informative-CDP conditions, assessing alignment with human examinees at the ability-distribution, mastery-profile, and item-difficulty levels. CDP improved all three levels: distributional overlap rose across configurations; weighted correlations between profile-level scores and human profile expectations reached 0.92 to 0.98; and item-difficulty recovery improved in rank order and absolute alignment, most for reasoning-enabled models; in the strongest case, Gemini 3.0 Flash (Thinking), one-parameter logistic (1PL) difficulty Spearman correlations rose from 0.24 to 0.86 and 0.90 and the root-mean-square error (RMSE) fell from 6.31 to 1.30 and 0.90; the informative condition helped most where profile-level alignment was strong. CDP brings LLM-simulated examinees into closer psychometric alignment with human examinees, making them practical for operational test development.
