Search papers, labs, and topics across Lattice.
The study introduces WrAFT, a modularized Writing Assessment and Feedback Tool designed to evaluate argumentative essays by providing accurate scores and comprehensive feedback. By leveraging various Large Language Models, including LLaMA-3.3-70B-Instruct and GPT-4o, the system achieves a state-of-the-art quadratic weighted kappa of 0.84 and an RMSE of 0.44 against official TOEFL scores. Human evaluations further validate its effectiveness, with high approval ratings for both surface-level and deep-level feedback, indicating its potential as a reliable tool for educational settings.
WrAFT achieves a remarkable 0.84 QWK in scoring essays, setting a new standard for automated writing evaluation systems.
This study presents WrAFT, a Writing Assessment and Feedback Tool, that delivers both accurate and reliable scores and effective comprehensive feedback to argumentative essays. WrAFT adopts a modular design by dividing automated writing evaluation (AWE) tasks into scoring, surface-level feedback, and deep-level feedback. In building the system, various Large Language Models (LLMs) have been evaluated, including LLaMA-3.3-70B-Instruct, GPT-4o, and Claude 3.7, through both direct prompting and supervised fine-tuning approaches. A proprietary dataset of 480 TOEFL Independent Writing essays with official benchmark scores was utilized. Benchmark-based evaluation shows that WrAFT achieves state-of-the-art performance in scoring, with a quadratic weighted kappa (QWK) of 0.84 and a root mean square error (RMSE) of 0.44 against official scores on a scale of 0-5. Human evaluation of system-generated feedback also reveals high approval ratings: 96.14 percent for surface-level feedback, 93.03 percent for deep-level macro feedback, and 94.69 percent for deep-level micro feedback. An interactive user interface has been developed for the system and is publicly available and free to use.
