Search papers, labs, and topics across Lattice.
This study critically evaluates four leading motion-based detectors for AI-generated videos, revealing that their performance is heavily influenced by preprocessing and sampling biases. The researchers found that these detectors are particularly sensitive to motion patterns unique to their training datasets, leading to a drastic drop in performance when tested on unbiased datasets. Notably, a frequency-based detection method demonstrated consistent effectiveness across varied datasets, highlighting a potential avenue for more robust detection strategies.
Motion-based detectors for AI-generated videos can fail dramatically when faced with unbiased datasets, exposing critical vulnerabilities in current evaluation methods.
The visual quality of AI-generated videos has improved drastically in recent years, making it increasingly difficult for humans to distinguish between real and synthetic media. In this work, we evaluate the robustness and applicability of four state-of-the-art motion-based AI-generated video detectors. We identify significant preprocessing and sampling biases in these methods and demonstrate that they account for a substantial portion of their reported performance. Furthermore, we find that these detectors are highly sensitive to motion patterns specific to their evaluation datasets, where AI-generated videos generally exhibit less inter-frame movement than real videos. We show that for all detectors, performance collapses to near-random levels when evaluated on a dataset that does not contain this motion bias. Additionally, through dataset rebalancing and the application of simple spatial augmentations, we observe severe performance degradation across all evaluated models. In contrast, we find that an existing frequency-based detector maintains strong performance across all evaluated datasets, suggesting that frequency-based approaches may offer a more generalizable path forward for AI-generated video detection. We hope that our work raises awareness towards these vulnerabilities and encourages the development of more representative, unbiased datasets and more robust evaluation protocols.
