Search papers, labs, and topics across Lattice.
This paper critiques the common practice of using random dataset splits for performance evaluation in AI, highlighting the issues of data leakage and hidden stratification in spatiotemporally correlated domains. To combat these issues, the authors introduce Structure-Aware Stratified Partitioning (SASP) for creating balanced validation splits and Curriculum Distributionally Robust Optimization (CDRO) to enhance training stability. Their framework demonstrates improved generalization and confidence calibration across various benchmarks, revealing failure modes obscured by traditional evaluation methods.
Data leakage and hidden stratification can inflate performance metrics, but a new framework reveals the true robustness of AI models in spatially correlated domains.
Performance evaluation in AI systems commonly assumes that random dataset splits produce independent and identically distributed (i.i.d.) subsets. We show that this assumption often breaks down in spatiotemporally correlated domains such as aerial surveillance, precision agriculture, and medical imaging, leading to two systematic failures: data leakage, where correlated samples span training and validation splits and inflate performance estimates, and hidden stratification, where errors on minority subpopulations are obscured by aggregate metrics. To address these issues, we propose a unified evaluation and training framework for spatially correlated data. We introduce Structure-Aware Stratified Partitioning (SASP), which constructs validation splits that reduce spatiotemporal leakage while preserving meaningful class balance, and Curriculum Distributionally Robust Optimization (CDRO), a curriculum-based relaxation of distributionally robust training that stabilizes optimization under these stricter splits. Across multiple benchmarks, this combination yields consistently improved generalization, more reliable confidence calibration, and exposes failure modes that remain hidden under conventional random-split evaluation.
