Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces "distinctiveness," a representation-level metric to evaluate learner representations by measuring how well they preserve separation between learners based on pairwise distances, without relying on labels or task-specific outcomes. They compare interaction-level representations (based on individual student questions) against learner-level representations (aggregating patterns across a student's interactions) using student-authored questions collected from a conversational AI agent. Results demonstrate that learner-level representations exhibit higher distinctiveness, stronger clustering, and more reliable pairwise discrimination, suggesting their superiority for differentiated modeling.
You can now evaluate student embeddings for AI education systems *before* training on downstream tasks, by measuring how well they separate individual learners.
Learner representations play a central role in educational AI systems, yet it is often unclear whether they preserve meaningful differences between students when instructional outcomes are unavailable or highly context-dependent. This work examines how to evaluate learner representations based on whether they retain separation between learners under a shared comparison rule. We introduce distinctiveness, a representation-level measure that evaluates how each learner differs from others in the cohort using pairwise distances, without requiring clustering, labels, or task-specific evaluation. Using student-authored questions collected through a conversational AI agent in an online learning environment, we compare representations based on individual questions with representations that aggregate patterns across a student's interactions over time. Results show that learner-level representations yield higher separation, stronger clustering structure, and more reliable pairwise discrimination than interaction-level representations. These findings demonstrate that learner representations can be evaluated independently of instructional outcomes and provide a practical pre-deployment criterion using distinctiveness as a diagnostic metric for assessing whether a representation supports differentiated modeling or personalization.