Search papers, labs, and topics across Lattice.
This paper addresses the challenge of detecting AI-generated videos by developing a multimodal detection system that integrates audio and visual analysis through an ensemble of models. By utilizing AASIST for audio detection and EfficientNet, XceptionNet, and MesoNet for visual feature extraction, the system combines predictions using mean averaging and stacking strategies to enhance robustness against diverse deepfake manipulations. The findings indicate that while individual models excel on specific datasets, the ensemble approach significantly improves generalization, achieving an average accuracy of around 70% across varied deepfake types.
Combining audio and visual analysis in a unified ensemble model boosts deepfake detection accuracy and generalization across diverse manipulations.
The increasing accessibility of artificial intelligence has led to a rapid rise in AI-generated videos, making it more difficult to distinguish between real and manipulated content. Many existing detection methods rely on a single model and often struggle to generalize across different types of deepfakes. In this work, we developed a multimodal deepfake detection system that combines both audio and visual analysis using an ensemble of models. The system includes AASIST for audio-based detection, and EfficientNet, XceptionNet, and MesoNet for analyzing visual features in video frames. The pipeline takes a video as input, separates the audio, and extracts face frames using MTCNN. Each model produces a score indicating the likelihood of the input being fake. These scores are then combined using ensemble strategies, including mean averaging and stacking. Mean fusion provides a simple and stable baseline, while stacking uses a trained meta-model to learn how to combine predictions more effectively. Results show that while individual models perform well on the datasets they were trained on, their performance drops when tested on more diverse datasets. The ensemble approach helps improve overall robustness by combining predictions from multiple models, leading to more consistent performance across different types of deepfakes. This suggests that using both audio and visual information together is a more reliable approach for deepfake detection. Our results highlight generalization to unseen manipulations as the central open challenge, with average accuracy around 70%.
