Search papers, labs, and topics across Lattice.
This paper investigates deep learning techniques for environmental sound deepfake detection (ESDD), distinguishing between fake and real audio recordings of sound scenes and events. The authors explore various spectrogram representations, network architectures (including pre-trained models), and ensemble methods to optimize ESDD performance. Results on benchmark datasets reveal that sound scene and sound event deepfake detection are distinct tasks, and fine-tuning pre-trained models like WavLM with a three-stage training strategy yields state-of-the-art performance, achieving high accuracy and F1 scores.
Separating sound scene and sound event deepfake detection as individual tasks dramatically improves performance, paving the way for more robust audio forensics.
In this paper, we propose a deep-learning framework for environmental sound deepfake detection (ESDD) -- the task of identifying whether the sound scene and sound event in an input audio recording is fake or not. To this end, we conducted extensive experiments to explore how individual spectrograms, a wide range of network architectures and pre-trained models, ensemble of spectrograms or network architectures affect the ESDD task performance. The experimental results on the benchmark datasets of EnvSDD and ESDD-Challenge-TestSet indicate that detecting deepfake audio of sound scene and detecting deepfake audio of sound event should be considered as individual tasks. We also indicate that the approach of finetuning a pre-trained model is more effective compared with training a model from scratch for the ESDD task. Eventually, our best model, which was finetuned from the pre-trained WavLM model with the proposed three-stage training strategy, achieve the Accuracy of 0.98, F1 Score of 0.95, AuC of 0.99 on EnvSDD Test subset and the Accuracy of 0.88, F1 Score of 0.77, and AuC of 0.92 on ESDD-Challenge-TestSet dataset.
