Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study evaluates large language models (LLMs) on a dataset of 32,534 double-marked GCSE mock exam responses, assessing their agreement with human examiners. The results reveal that LLMs not only align closely with examiner consensus but also outperform the inter-examiner agreement, particularly excelling in subjective assessments like English essays and complex handwritten math scripts. This finding suggests that LLMs can serve as reliable and cost-effective alternatives for automated exam marking across various subjects.
LLMs achieve higher agreement with examiners than the examiners do with each other, challenging traditional assumptions about human grading reliability.
We introduce a dataset of 32,534 double-marked real student responses to GCSE mock exams (GCSEs are the UK's national exams, taken at age ~16), spanning 328 questions across five subjects and including handwritten work. We test whether off-the-shelf large language models agree with examiners as closely as the two examiners agree with each other. We find that models overwhelmingly agree well with the examiner consensus across subjects, with the top performing models agreeing more closely with examiners than examiners agree with each other. Models achieve high scores for subjective tasks like English essay marking, as well as handling complex and messy handwritten Maths paper scripts. Agreement is uniform near the examiner line, and not massively discriminated by model size, providing cost-effective automated marking solutions.