Search papers, labs, and topics across Lattice.
This paper introduces DF-MoE, a novel framework for deepfake detection that leverages a Mixture-of-Experts architecture to integrate multiple high-level cues from audio and visual modalities. By utilizing a diverse set of pre-trained models to extract features related to facial movements, expressions, and audio characteristics, DF-MoE effectively addresses the challenge of generalization across various deepfake generation methods. Experimental results demonstrate that DF-MoE outperforms existing state-of-the-art techniques across five benchmark datasets, highlighting its robustness and effectiveness in real-world applications.
DF-MoE achieves superior deepfake detection performance by harnessing a diverse set of multimodal features, setting a new standard in the field.
Audio-visual deepfake detection is an actively studied topic, where one of the main challenges is to develop detectors able to generalize across deepfake generation methods. We conjecture that overfitting can be mitigated by extracting multiple high-level cues from the available audio and visual modalities via pre-trained models. We therefore assemble a wide variety of pre-trained models to extract features that encode mouth movements, face parsing, facial expressions, head pose, gaze tracking, heart rate, audio emotion and speech activity. We further integrate both unimodal and multimodal cues via a Mixture-of-Experts (MoE) backbone to detect deepfakes. We perform in-domain and cross-domain experiments on five benchmarks for deepfake detection (MAVOS-DD, AVLips, PolyGlotFake, BioDeepAV, FakeAVCeleb) to compare our framework (DF-MoE) with state-of-the-art methods. Our results indicate that DF-MoE obtains superior deepfake detection results, surpassing all competing methods. We release our code at https://github.com/vladhondru25/DF-MoE.
