VendorBench-100: A Unified Cross-Paradigm Benchmark for Deepfake Image Detection

View on arXiv ← Back to list

Authors: Sharayu N. Deshmukh, Md Rashidunnabi, Nelton Tiago Gemo, Kurundkar G. D., Mahamune M. R., Nilesh K. Deshmukh

Published: 2026-07-07 13:22:00+00:00

Comment: 22 pages, 10 figures, 3 tables. Code and data: https://github.com/sharayu-20/vendorbench-100

AI Summary

This paper introduces VendorBench-100, a cross-paradigm benchmark for deepfake image detection that evaluates 36 models from commercial APIs, zero-shot vision-language models (LLMs), and open-source detectors using a single, adversarial 100-image corpus. The benchmark emphasizes challenging real-world scenarios and uses Matthews correlation coefficient (MCC) and ROC-AUC for evaluation. The study reveals that commercial APIs generally perform best, but a significant divergence exists between a model's ranking ability (ROC-AUC) and its operating-point quality (MCC), which is considered the central finding.

Abstract

Deepfake image detection is currently served by three fundamentally different paradigms: commercial APIs, zero-shot vision-language models (LLMs), and open-source detectors. Despite their widespread use, these paradigms are rarely evaluated under a common protocol, making direct comparison difficult. We introduce VendorBench-100, a cross-paradigm benchmark that evaluates 36 representative models using a single adversarial 100-image corpus, a unified output schema, and a common evaluation framework. To ensure reliable assessment under the corpus's intentional class imbalance, models are ranked primarily by the Matthews correlation coefficient (MCC), with ROC-AUC reported as a threshold-independent measure of ranking ability. Rather than maximizing dataset size, VendorBench-100 emphasizes challenging real-world scenarios through a curated taxonomy of eight edge-case families, including face swaps, text-to-video stills, AI photo edits, avatar compositing, opaque-provenance images, and compressed research frames. Our evaluation shows that commercial APIs achieve the strongest median performance, followed by vision LLMs and open-source detectors. However, individual open-source models remain competitive with the best vision LLMs. More importantly, we identify a consistent divergence between ranking ability (ROC-AUC) and operating-point quality (MCC), demonstrating that strong score discrimination does not necessarily produce reliable default-threshold decisions. This metric disagreement, rather than any single leaderboard ranking, is the central finding of the benchmark. We release the complete evaluation framework and benchmark results to support reproducible future research. The source code and data are available at: https://github.com/sharayu-20/vendorbench-100

Key findings

Commercial APIs generally achieve the strongest median performance, followed by vision LLMs, and then open-source detectors. A consistent and significant divergence was found between models' ranking ability (ROC-AUC) and their operating-point quality (MCC), indicating that strong discrimination does not guarantee reliable default-threshold decisions. This divergence, rather than any single leaderboard ranking, is highlighted as the central finding, stressing the importance of evaluating both metrics for practical deployment.

Approach

They developed VendorBench-100, a unified benchmark for deepfake image detection. It consists of 100 adversarial images, a common output schema, and an evaluation framework that ranks models primarily by Matthews correlation coefficient (MCC) and secondarily by ROC-AUC to account for class imbalance and distinguish between ranking ability and default-threshold decisions.

Datasets

VendorBench-100 (a custom-built adversarial 100-image corpus with 79 fake and 21 real images, drawing on over 20 provenance sources, including FaceForensics++ and DF40 style frames).

Model(s)

Commercial APIs (Neural Defend, Reality Defender, Sightengine, TruthScan, Hive AI), Vision LLMs (Gemini, Claude Opus 4.8, Qwen, Llama 4 Maverick, Nemotron vision-language variant, GPT, GLM-5.2), Open-source detectors (DRCT, RINE, C2P-CLIP, DeCLIP, AIDE, GenD-DINOv3-L, Community Forensics ViT-384, ntire2026_deepfake, dima806_ai_vs_real, bombek1_siglip_dinov2, organika_sdxl, aidfr_real_v2, nahrawy_aiornot, ummmaybe_vit, yaya_source, ash_flux_vit, king1oo1_deepguard, haywoodsloan_deploy, date3k2_vit, jacob_distilled, opensight_commfor, ateeqq_siglip2, sadra_sdxl_face, wvolf_vit).

Author countries

Portugal, India

← Previous