Search papers, labs, and topics across Lattice.
This paper critiques the conventional use of Area Under the ROC Curve (AUC) for evaluating deepfake detectors, arguing that it inadequately reflects real-world challenges posed by diverse data sources and artifact types. To address this, the authors introduce a novel metric called Cross-dataset AUC (Cross-AUC), which incorporates prediction polarization by averaging per-domain AUCs and quantifying distribution shifts using Wasserstein Distance. Experiments across seven benchmark datasets reveal that Cross-AUC provides a more accurate and interpretable assessment of detector robustness under domain shifts, highlighting its practical significance in evaluating deepfake detection systems.
Current AUC metrics can mislead deepfake detector evaluations, but Cross-AUC reveals the true impact of domain shifts on performance.
Recent advances in generative AI, such as diffusion models and face-swapping tools, have enabled the creation of highly realistic deepfakes, leading to real-world harms including financial fraud and non-consensual explicit content. In response, deepfake detection has become an active research area, with recent methods increasingly focusing on improving generalization to unseen manipulations. This is typically evaluated using the Area Under the ROC Curve (AUC) measured separately across multiple datasets. However, such an evaluation fails to reflect real-world scenarios where detectors face a mixture of data sources and varying artifact types. To address this limitation, we introduce a novel metric, Cross-dataset AUC (Cross-AUC) that averages per-domain AUCs with a measure of prediction polarization for taking into account the robustness to domain shift. The polarization extent is quantified by the Wasserstein Distance between class score distributions. Cross-AUC not only assesses the generalization capabilities of deepfake detectors under domain shifts more realistically, but it is also interpretable as it better explains the reason behind a drop in performance. Experiments performed on seven benchmark datasets demonstrate its practical relevance.
