Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study introduces a conditional generalizability framework to assess the reliability of automated essay scoring systems, addressing the variability in measurement adequacy across different response conditions. By comparing analytical D-study projections with empirical configuration sweeps, the authors reveal important insights into the adequacy of scoring designs, particularly when conditioned on entropy-defined response strata. The findings indicate that while overall dependability remains high, it varies across different strata, suggesting tailored decision-study requirements for optimal scoring configurations.
Dependability in automated essay scoring drops significantly across different response conditions, revealing the need for tailored evaluation strategies.
Aggregate reliability estimates can obscure heterogeneity in measurement-design burden across response conditions, so a single G- or D-study may mischaracterize a design's adequacy for particular strata. This study introduces a conditional generalizability framework with three components. First, automated scoring configurations -- the encoder architectures and scoring-head families admissible within a fixed pipeline -- are treated as a universe of admissible measurement conditions rather than incidental modeling choices. Second, analytical D-study projections are compared with empirical configuration sweeps over a finite scoring pool, yielding two estimands of design adequacy whose agreement or divergence diagnoses the realized configuration universe. Third, evidence is conditioned on entropy-defined response strata, treating entropy as an operational stratification variable, not a construct claim about writing quality. Whereas recent generalizability-theory extensions address AI-generated item variants on the response side, this framework addresses the analogous scoring-side problem: AI-mediated scoring configurations. Demonstrated with automated essay scoring of timed L2 writing, the realized design was dependable in aggregate (Phi approx 0.76). Re-estimated within entropy strata, dependability stayed high but declined modestly and robustly (Phi = 0.88, 0.87, 0.84) -- a gradient implying different decision-study requirements, the highest-entropy stratum requiring the most crossed conditions. The framework offers a portable workflow for evaluating nonuniform dependability.