Search papers, labs, and topics across Lattice.
This paper introduces an LLM-based pipeline for tagging learning resources with competencies, addressing the challenge of manual tagging in Learning Management Systems. The pipeline segments resources, retrieves candidate competencies using BM25 and graph context, and uses an LLM to select relevant competencies with supporting evidence. Experiments on a computer science competency dataset demonstrate that the proposed LLM+BM25+Graph (LBG) pipeline outperforms various baselines, achieving a micro-F1 of 0.57 and macro-F1 of 0.50 at the fragment level, while providing traceable evidence spans.
Forget brittle, opaque tagging systems: this LLM pipeline aligns learning resources to competencies with high accuracy *and* provides human-auditable evidence for each tag.
Linking learning resources to a structured competency framework is key to enabling competency-based search and curriculum analytics in Learning Management Systems (LMS). However, manual tagging is labor-intensive, and fully automatic methods often lack transparency. In this paper, we present an end-to-end alignment pipeline that uses a large language model (LLM) as a constrained, evidence-producing tagger. LMS resources -both instructional content and assessments -are first segmented into meaningful pedagogical fragments. For each fragment, a small set of candidate competencies is retrieved from structured competency profiles enriched with graph-based context. The LLM then selects the most relevant competencies from this set and provides supporting evidence spans from the fragment text. These predictions are refined using the structure of the competency graph and aggregated at the resource level. We evaluate our approach on a dataset built from the Computer Science department's competency referential at the Universit茅 de Technologie de Compi猫gne (UTC), covering 22 competencies across multiple course materials. Our LLM+BM25+Graph (LBG) pipeline achieves strong results, with a micro-F1 of 0.57 and macro-F1 of 0.50 at the fragment level, 0.51 macro-F1 at the resource level, and an MRR of 0.82outperforming zero-shot and few-shot LLM variants, retrieval/similarity baselines, and supervised classifiers -while also producing more mechanically traceable evidence spans to support human auditing and educational analysis.
