Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study introduces AiAWE, an open-source automated writing evaluation system that leverages a LoRA-adapted instruction-tuned large language model, Gemma-3-27B-it, to score argumentative essays. Evaluated on a proprietary dataset of TOEFL Independent Writing essays, the fine-tuned Gemma model outperformed both the larger LLaMA-3.3-70B and the fine-tuned GPT-3.5 baseline, achieving a root mean square error of 0.474 and a quadratic weighted kappa of 0.828. Key findings reveal that open-weight LLMs can achieve superior performance compared to proprietary models, model size does not guarantee better performance, and identical LoRA configurations can lead to different adaptation outcomes across architectures.
Open-weight LLMs can outperform proprietary models in automated essay scoring, challenging assumptions about model size and adaptation consistency.
This study presents AiAWE, an open-source automated writing evaluation system that scores argumentative essays using a LoRA-adapted instruction-tuned large language model (Gemma-3-27B-it). Using a proprietary Educational Testing Service (ETS) dataset of 480 TOEFL Independent Writing essays, we fine-tune Gemma-3-27B and LLaMA-3.3-70B under identical LoRA configurations on a 120-essay training subset and evaluate on the remaining 360 essays under identical inference quantization. The fine-tuned Gemma model achieves a root mean square error of 0.474, a quadratic weighted kappa of 0.828, and an agreement rate of 90.56% within +/- 0.5 of the human score, outperforming both the larger LLaMA-3.3-70B model and the fine-tuned GPT-3.5 baseline reported in prior work on the same dataset. Three findings are of broader interest: open-weight LLMs can match or exceed proprietary fine-tuning for rubric-aligned scoring; model scale is not a reliable predictor of downstream performance under LoRA adaptation; and identical LoRA hyperparameters produce qualitatively different adaptation behaviors across architectures. The production system runs on a consumer-grade server and is publicly accessible at https://app.awade.gec.waseda.ac.jp. LoRA adapters, application code, and fine-tuning YAMLs are publicly available through their respective repositories.