Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper derives two Quadratic Weighted Kappa (QWK) ceilings for Automated Essay Scoring (AES) based on Classical Test Theory: a theoretical ceiling representing perfect prediction of latent true scores under label noise, and a human-like ceiling representing human-level scoring error. They demonstrate that human-human QWK underestimates the true ceiling and show how these ceilings clarify the current performance and remaining headroom of modern AES models. Simulation and real-world benchmark experiments validate the proposed ceilings.
Human agreement, often used as a ceiling for AES performance, significantly underestimates the true potential accuracy.
Automated essay scoring (AES) is commonly evaluated on public benchmarks using quadratic weighted kappa (QWK). However, because benchmark labels are assigned by human raters and inevitably contain scoring errors, it remains unclear both what QWK is theoretically attainable and what level is practically sufficient for deployment. We therefore derive two dataset-specific QWK ceilings based on the reliability concept in classical test theory, which can be estimated from standard two-rater benchmarks without additional annotation. The first is the theoretical ceiling: the maximum QWK that an ideal AES model that perfectly predicts latent true scores can achieve under label noise. The second is the human-like ceiling: the QWK attainable by an AES model with human-level scoring error, providing a practical target when AES is intended to replace a single human rater. We further show that human--human QWK, often used as a ceiling reference, can underestimate the true ceiling. Simulation experiments validate the proposed ceilings, and experiments on real benchmarks illustrate how they clarify the current performance and remaining headroom of modern AES models.