Search papers, labs, and topics across Lattice.
This study evaluates the grading consistency of two large language models (LLMs), Grok and GPT, in the context of graduate-level software engineering courses, using a dataset of 180 student submissions. The research uncovers significant inter-model inconsistencies and reveals that continuous interaction history can lead to systematic drift in LLM grading standards away from human expert evaluations. These findings highlight the potential of LLMs to alleviate grading burdens while cautioning against their indiscriminate use, as it may exacerbate educational inequities without tailored operational practices.
Continuous interaction with LLMs can skew grading standards, leading to systemic unfairness in academic assessments.
Graduate-level research reading report assessment creates a substantial labor burden for educators. While large language models (LLMs) hold great potential for automating academic grading, their reliability for this specialized task remains understudied, particularly regarding grading consistency, the lack of which represents a primary obstacle to educational fairness. This paper proposes a human-aligned LLM-assisted grading workflow and presents a case study based on 180 student submissions from a graduate advanced software engineering course. We evaluate two mainstream LLMs, Grok and GPT, in terms of grading consistency and alignment with human scores. We find LLMs exhibit distinct levels of intra-model consistency and significant inter-model grading inconsistencies, while simple ensemble approaches cannot improve alignment with human evaluation. Critically, continuous interaction history drives systematic drift in models' grading standards away from human expert scores. Our findings demonstrate LLMs' potential in reducing grading workload for educators in graduate education, while highlighting that indiscriminate LLM grading may introduce systemic unfairness, suggesting that specific operational practices are required to mitigate such disparities.
