Search papers, labs, and topics across Lattice.
This paper conducts a comprehensive audit of 39 deepfake speech datasets, assessing their accessibility, documentation, demographic representation, and scale. The findings highlight critical limitations in fairness assessments due to the lack of demographic metadata, which hampers subgroup analysis and obscures the datasets' representativeness. Additionally, the significant overlap in bona fide speech sources across datasets raises concerns about the validity of cross-dataset evaluations and the potential for inflated generalization claims in deepfake speech detection research.
Fairness assessments of deepfake speech detectors are fundamentally flawed due to a lack of demographic metadata in most datasets, limiting meaningful subgroup analysis.
Claims about the robustness and fairness of deepfake speech detectors are only as credible as the datasets used to train and evaluate those systems. We present a dataset-level audit of the deepfake speech landscape. We compile and analyze 39 deepfake speech datasets, examining key attributes including accessibility, documentation, demographic and language coverage, dataset scale, and the underlying bona fide speech sources. Our audit reveals two important takeaways. Firstly, fairness assessment is largely infeasible because most datasets lack demographic metadata, and only a few contain gender or language labels. This prevents any meaningful subgroup analysis and leaves other demographic attributes unaddressed. Secondly, we identify substantial overlap in underlying bona fide source corpora across datasets, which can undermine cross-dataset evaluation and lead to overstated generalization claims.
