Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces TR-EduVSum, a new dataset of 82 Turkish educational videos with 3281 human-generated summaries, and proposes AutoMUP, a novel method for automatically generating gold-standard summaries from multiple human summaries. AutoMUP leverages embedding-based clustering of meaning units and statistical modeling of inter-participant agreement to create consensus-weighted summaries. Experiments demonstrate that AutoMUP summaries achieve high semantic overlap with LLM-generated summaries and that consensus weighting and clustering are crucial for summary quality.
Forget expensive human annotation: this framework automatically generates gold-standard summaries of educational videos by distilling consensus from multiple human-generated summaries.
This study presents a framework for generating the gold-standard summary fully automatically and reproducibly based on multiple human summaries of Turkish educational videos. Within the scope of the study, a new dataset called TR-EduVSum was created, encompassing 82 Turkish course videos in the field of "Data Structures and Algorithms" and containing a total of 3281 independent human summaries. Inspired by existing pyramid-based evaluation approaches, the AutoMUP (Automatic Meaning Unit Pyramid) method is proposed, which extracts consensus-based content from multiple human summaries. AutoMUP clusters the meaning units extracted from human summaries using embedding, statistically models inter-participant agreement, and generates graded summaries based on consensus weight. In this framework, the gold summary corresponds to the highest-consensus AutoMUP configuration, constructed from the most frequently supported meaning units across human summaries. Experimental results show that AutoMUP summaries exhibit high semantic overlap with robust LLM (Large Language Model) summaries such as Flash 2.5 and GPT-5.1. Furthermore, ablation studies clearly demonstrate the decisive role of consensus weight and clustering in determining summary quality. The proposed approach can be generalized to other Turkic languages at low cost.