Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper enhances the argument-based validation (ABV) framework for Automated Essay Scoring (AES) by integrating fairness analysis, linguistic feature correlations, prediction error evaluation, and model agreement with human raters. By applying this refined framework to evaluate eight model architectures on a large corpus of French exam essays, the authors reveal critical insights into model performance and generalizability. The findings underscore the importance of multidimensional assessments in high-stakes language testing, advancing the field of French AES significantly.
AES models can significantly misalign with human judgment, revealing critical gaps in their reliability and fairness.
In Automated Essay Scoring (AES), benchmarking practices have fostered minimalist evaluation practices, in contrast with the broader-view recommendations of evaluation frameworks, such as the argument-based validation framework (ABV), which argued in favor of a multidimensional assessment of systems, especially in the context of high-stakes language tests. In this paper, we introduce an enhanced and more practical version of the ABV framework, incorporating fairness analysis, correlations with linguistic features, prediction error evaluation, and model agreement compared with human raters. Applying this framework to French AES, we compare 8 model architectures on a corpus of 27k exam essays (2 raters each) and a generalization corpus of 961 essays (at least nine raters each). Our analyses illustrate the benefits of applying the ABV framework to better understand the capabilities and pitfalls of AES models, while also advancing the state-of-the-art for French AES.