Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The paper investigates the stochastic variance introduced by subject-exclusive cross-validation in facial Action Unit (AU) detection, revealing a significant noise floor in commonly used metrics like F1 score. Through repeated 3-fold cross-validation on BP4D+, they quantify this variance and demonstrate that model ranking is unstable across different fold assignments. They advocate for Leave-One-Dataset-Out (LODO) evaluation, showing it exposes domain-level instability masked by single-dataset cross-validation and provides more reliable results.
Cross-validation's inherent randomness can make reported improvements in facial action unit detection illusory, demanding a shift to more robust Leave-One-Dataset-Out evaluation.
Subject-exclusive cross-validation is the standard evaluation protocol for facial Action Unit (AU) detection, yet reported improvements are often small. We show that cross-validation itself introduces measurable stochastic variance. On BP4D+, repeated 3-fold subject-exclusive splits produce an empirical noise floor of $\pm 0.065$ in average F1, with substantially larger variation for low-prevalence AUs. Operating-point metrics such as F1 fluctuate more than threshold-independent measures such as AUC, and model ranking can change under different fold assignments. We further evaluate cross-dataset robustness using a Leave-One-Dataset-Out (LODO) protocol across five AU datasets. LODO removes partition randomness and exposes domain-level instability that is not visible under single-dataset cross-validation. Together, these results suggest that gains often reported in cross-fold validation may fall within protocol variance. Leave-one-dataset-out cross-validation yields more stable and interpretable findings