Search papers, labs, and topics across Lattice.
This paper introduces AIriskEval-edu-db2, a novel dataset designed to enhance the training and evaluation of auditors using LLMs for assessing pedagogical risks in K-12 educational content. The dataset features 1,639 explanations from curated ScienceQA questions, including human-generated explanations and LLM-simulated profiles, annotated for five dimensions of risk. Validation experiments reveal that a lightweight local Llama 3.1 8B model, fine-tuned on this dataset, can match or exceed the performance of state-of-the-art models in detecting pedagogical risks while ensuring privacy in educational assessments.
A lightweight model fine-tuned on AIriskEval-edu-db2 can rival leading models in pedagogical risk detection, all while maintaining privacy in educational settings.
This work introduces AIriskEval-edu-db2, a new dataset designed to train and evaluate auditors based on LLMs for an explainable pedagogical risk assessment in instructional content for grades K-12. The dataset comprises 1,639 explanations from 170 curated ScienceQA questions, covering science, language arts, and social sciences. For each question, the dataset includes an explanation written by a human teacher alongside 11 explanations generated by LLM-simulated teacher profiles associated with distinct pedagogical risks. We propose a comprehensive risk rubric aligned with established educational standards that covers five complementary dimensions: factual precision, depth and completeness, focus and relevance, student-level appropriateness, and ideological bias. A key contribution is the addition of 785 explanations with structured explainability annotations, including risk localization and risk description. The annotations are produced through a semi-automatic process with expert teacher validation. Finally, we present validation experiments comparing state-of-the-art proprietary models with a lightweight local Llama 3.1 8B model in both the pedagogical risk detection and the explainability assessment. These experiments evaluate whether supervised fine-tuning on AIriskEval-edu-db2 enables a locally deployable model to approach or outperform stronger frontier models while preserving privacy in educational auditing and assessment tasks.
