Search papers, labs, and topics across Lattice.
This paper introduces SEFORA, a comprehensive public corpus that pairs instructor feedback with student writing assignments, addressing the lack of real-world data on feedback delivery in educational settings. To evaluate the effectiveness of LLM-generated feedback, the authors develop UniMatch, a reference-based evaluation framework that quantifies the alignment of machine-generated feedback with instructor standards. The findings indicate that across various LLMs, no configuration achieves an F1 score above 0.4, highlighting significant challenges in aligning AI-generated feedback with instructor priorities, especially as feedback length increases.
LLMs struggle to replicate the nuanced feedback that instructors prioritize, with performance dropping as feedback complexity rises.
Effective writing feedback is among the strongest drivers of student learning, yet producing it at scale is labor-intensive. LLMs offer a natural path to scaling writing support, but two gaps stand in the way: few public corpora capture how instructors actually deliver feedback in real classrooms, and no reliable method measures whether generated feedback aligns with what an instructor would write. We address both. SEFORA is a public corpus pairing instructor inline feedback with assignment prompts, rubrics, scores, and multi-draft revisions across various college writing genres, comprising 564 drafts and 8,240 instructor annotations. UniMatch is a reference-based evaluation framework for open-ended generation: it segments feedback into feedback units, scores their semantic correspondence under instructor-derived criteria, and aligns them via optimal matching to yield interpretable precision, recall, and F1. Across 74 experimental configurations spanning multiple LLMs, no setting exceeds 0.4 F1. UniMatch reveals that models struggle to identify the feedback instructors would prioritize, and performance degrades as models generate more.
