Search papers, labs, and topics across Lattice.
This paper formalizes the problem of temporal leakage in early outcome prediction models built from Learning Management System (LMS) logs. They introduce LEAP (Leakage-Excluded Early-Availability Protocol) to enforce cutoff-first truncation and audit feature provenance, preventing post-cutoff evidence from influencing the benchmark. Applying LEAP to the OULAD dataset reveals that temporal leakage, particularly through assessment information, significantly inflates reported "early" performance, and that Random Forest performs best at early cutoffs while Gradient Boosting dominates later.
Naive early prediction models can be dangerously overconfident: temporal leakage from assessment data can dramatically inflate reported performance.
Early-warning models built from Learning Management System (LMS) logs aim to predict end-of-course outcomes early enough to enable timely learner support. However, reported "early" performance is often inflated by temporal leakage. This occurs when the pipeline uses information that would not yet be available at the time of prediction. We formalize cutoff-based early outcome prediction under a temporal availability constraint and introduce LEAP (Leakage-Excluded Early-Availability Protocol), which enforces cutoff-first truncation prior to joins and aggregation and audits feature provenance to prevent post-cutoff evidence from entering the benchmark. We instantiate LEAP on the public Open University Learning Analytics Dataset (OULAD) as a multi-step protocol for leakage-controlled evaluation across weekly cutoffs. Using several standard learning methods, we evaluate performance using ROC-AUC, PR-AUC, Brier score, and F1@0.5. Results show improving performance as the observation window expands, with a marked gain around week~3; Random Forest performs best at the earliest cutoffs, while Gradient Boosting dominates thereafter. Leakage ablations further show that temporal violations, especially through assessment information, can inflate apparent "early" performance.
