Search papers, labs, and topics across Lattice.
This paper evaluates LLM-as-a-judge performance across physics assessments (structured questions, essays, plots) using GPT-5.2, Grok 4.1, Claude Opus 4.5, DeepSeek-V3.2, and Gemini Pro 3. Results show high validity for structured questions and plots, especially with provided solutions, but poor validity for essay marking, even with mark schemes or exemplar anchoring. The study concludes that criterion-referenceability, not just model capability, determines the trustworthiness of LLM marking.
LLMs can ace grading physics problems with clear solutions, but fall flat when judging essays, revealing that their assessment skills hinge more on task clarity than raw intelligence.
As large language models (LLMs) are increasingly considered for automated assessment and feedback, understanding when LLM marking can be trusted is essential. We evaluate LLM-as-a-judge marking across three physics assessment formats - structured questions, written essays, and scientific plots - comparing GPT-5.2, Grok 4.1, Claude Opus 4.5, DeepSeek-V3.2, Gemini Pro 3, and committee aggregations against human markers under blind, solution-provided, false-solution, and exemplar-anchored conditions. For $n=771$ blind university exam questions, models achieve fractional mean absolute errors (fMAE) $\approx 0.22$ with robust discriminative validity (Spearman $蟻> 0.6$). For secondary and university structured questions ($n=1151$), providing official solutions reduces MAE and strengthens validity (committee $蟻= 0.88$); false solutions degrade absolute accuracy but leave rank ordering largely intact (committee $蟻= 0.77$; individual models $蟻\geq 0.59$). Essay marking behaves fundamentally differently. Across $n=55$ scripts ($n=275$ essays), blind AI marking is harsher and more variable than human marking, with discriminative validity already poor ($蟻\approx 0.1$). Adding a mark scheme does not improve discrimination ($蟻\approx 0$; all confidence intervals include zero). Anchored exemplars shift the AI mean close to the human mean and compress variance below the human standard deviation, but discriminative validity remains near-zero - distributional agreement can occur without valid discrimination. For code-based plot elements ($n=1400$), models achieve exceptionally high discriminative validity ($蟻> 0.84$) with near-linear calibration. Across all task types, validity tracks criterion-referenceability - the extent to which a task maps to explicit, observable grading features - and benchmark reliability, rather than raw model capability.
