Search papers, labs, and topics across Lattice.
This paper introduces VendorBench-100, a comprehensive benchmark designed to evaluate deepfake image detection across three distinct paradigms: commercial APIs, zero-shot vision-language models (LLMs), and open-source detectors. By employing a curated 100-image corpus and a unified evaluation framework, the study reveals that while commercial APIs outperform other models on average, individual open-source detectors can compete effectively with leading vision LLMs. A key finding is the significant divergence between ranking ability and operating-point quality, highlighting that high ROC-AUC scores do not guarantee reliable threshold decisions, which is critical for real-world applications.
A striking divergence between ranking ability and operating-point quality reveals that high ROC-AUC scores can mislead deepfake detection effectiveness.
Deepfake image detection is currently served by three fundamentally different paradigms: commercial APIs, zero-shot vision-language models (LLMs), and open-source detectors. Despite their widespread use, these paradigms are rarely evaluated under a common protocol, making direct comparison difficult. We introduce VendorBench-100, a cross-paradigm benchmark that evaluates 36 representative models using a single adversarial 100-image corpus, a unified output schema, and a common evaluation framework. To ensure reliable assessment under the corpus's intentional class imbalance, models are ranked primarily by the Matthews correlation coefficient (MCC), with ROC-AUC reported as a threshold-independent measure of ranking ability. Rather than maximizing dataset size, VendorBench-100 emphasizes challenging real-world scenarios through a curated taxonomy of eight edge-case families, including face swaps, text-to-video stills, AI photo edits, avatar compositing, opaque-provenance images, and compressed research frames. Our evaluation shows that commercial APIs achieve the strongest median performance, followed by vision LLMs and open-source detectors. However, individual open-source models remain competitive with the best vision LLMs. More importantly, we identify a consistent divergence between ranking ability (ROC-AUC) and operating-point quality (MCC), demonstrating that strong score discrimination does not necessarily produce reliable default-threshold decisions. This metric disagreement, rather than any single leaderboard ranking, is the central finding of the benchmark. We release the complete evaluation framework and benchmark results to support reproducible future research. The source code and data are available at: https://github.com/sharayu-20/vendorbench-100
