Search papers, labs, and topics across Lattice.
This paper evaluates instruction-tuned LLMs on essay scoring across three datasets (ASAP 2.0, ELLIPSE, DREsS), comparing their performance against human scores under both holistic and analytic rubrics. The study finds moderate to high agreement on holistic scoring but significant negative bias on Lower-Order Concern (LOC) traits like grammar in analytic scoring. It also demonstrates that concise keyword prompts outperform longer rubric-style prompts for multi-trait analytic scoring and quantifies the data needed to detect biases.
LLMs consistently mark down grammar and conventions more harshly than human graders, revealing a systematic bias that can be detected with surprisingly little data.
Despite growing interest in using Large Language Models (LLMs) for educational assessment, it remains unclear how closely they align with human scoring. We present a systematic evaluation of instruction-tuned LLMs across three open essay-scoring datasets (ASAP 2.0, ELLIPSE, and DREsS) that cover both holistic and analytic scoring. We analyze agreement with human consensus scores, directional bias, and the stability of bias estimates. Our results show that strong open-weight models achieve moderate to high agreement with humans on holistic scoring (Quadratic Weighted Kappa about 0.6), but this does not transfer uniformly to analytic scoring. In particular, we observe large and stable negative directional bias on Lower-Order Concern (LOC) traits, such as Grammar and Conventions, meaning that models often score these traits more harshly than human raters. We also find that concise keyword-based prompts generally outperform longer rubric-style prompts in multi-trait analytic scoring. To quantify the amount of data needed to detect these systematic deviations, we compute the minimum sample size at which a 95% bootstrap confidence interval for the mean bias excludes zero. This analysis shows that LOC bias is often detectable with very small validation sets, whereas Higher-Order Concern (HOC) traits typically require much larger samples. These findings support a bias-correction-first deployment strategy: instead of relying on raw zero-shot scores, systematic score offsets can be estimated and corrected using small human-labeled bias-estimation sets, without requiring large-scale fine-tuning.
