Search papers, labs, and topics across Lattice.
This study evaluates the effectiveness of human detection of deepfake audio across three speech synthesis tools from 2019, 2022, and 2024, revealing a significant decline in detection accuracy, particularly with the ElevenLabs synthesizer. Despite warnings about deepfakes, human detection accuracy for fully synthetic speech dropped to an F1 score of 48%, and accuracy for partial spoofing plummeted to just 9%. The research highlights the complementary failures of human listeners and pretrained detectors, emphasizing the urgent need for enhanced verification methods to maintain trust in audio authenticity.
Human listeners misclassify genuine audio as fake 77% of the time, revealing a critical vulnerability in our ability to discern deepfake speech.
Advances in speech synthesis have made deepfake audio highly realistic. Earlier studies reported 70-80% human detection accuracy, but relied primarily on older synthesizers. We compare human detection for three selected voice synthesis tools released in 2019, 2022, and 2024 with 82 IT professionals, and benchmark humans against six pretrained detectors on the same material. For fully synthetic speech (full spoofs), the F1 score drops from about 90% for RTVC and YourTTS to 48% for ElevenLabs, although listeners were explicitly warned that deepfakes were present. For partial spoofing, where only one sentence of an utterance is altered, strict accuracy falls to 9%, and listeners classify the synthetic sentence as bona fide 77% of the time. Humans and detectors fail in complementary ways, and neither reliably localizes short manipulations. Additionally, listeners increasingly mislabel bona fide speech as fake, eroding trust in unmanipulated audio. These findings show that human perception alone is unreliable for the selected modern and partial-spoof conditions and motivate procedural verification, provenance, watermarking, and segment-level detection.
