Search papers, labs, and topics across Lattice.
The paper explores detecting deepfake speech by identifying manipulated words within an utterance using a fine-tuned Whisper model. This approach leverages next-token prediction during transcription to identify synthetic words, offering a cost-effective alternative to dedicated synthetic word detectors. Experiments show that the fine-tuned Whisper achieves competitive performance compared to a ResNet-based detector, particularly on in-domain data, while also providing transcription.
Repurpose Whisper as a deepfake detector and you can transcribe and spot synthetic words in one pass, rivaling dedicated models.
Deepfake speech utterances can be forged by replacing one or more words in a bona fide utterance with semantically different words synthesized by speech generative models. While a dedicated synthetic word detector could be developed, we investigate a cost-effective method that fine-tunes a pre-trained Whisper model to detect synthetic words while transcribing the input utterance via next-token prediction. We further investigate using partially vocoded utterances as the fine-tuning data, thereby reducing the cost of data collection. Our experiments demonstrate that, on in-domain test data, the fine-tuned Whisper yields low synthetic-word detection error rates and transcription error rates. On out-of-domain test data with synthetic words produced by unseen speech generative models, the fine-tuned Whisper remains on par with a dedicated ResNet-based detection model; however, the overall performance degradation calls for strategies to improve its generalization capability.
