Search papers, labs, and topics across Lattice.
This paper introduces the Calibrated Deepfake Trust Score (CDTS), a novel framework that reframes deepfake detection as a calibrated trust instrument, emphasizing the importance of calibration in addition to accuracy. The authors establish a strong correlation between the competence of deepfake detectors and the degradation of trust calibration, finding that as detector competence decreases, trustworthiness also diminishes significantly across various architectures. Their findings reveal that a competence-aware approach not only enhances calibration but also addresses inequities in trust across demographic subgroups, making the CDTS a crucial tool for improving deepfake detection systems.
Trust in deepfake detection systems can plummet as their competence wanes, revealing a critical link between performance and calibration that could redefine how we assess AI trustworthiness.
Modern deepfake detectors are rarely consumed as bare classifiers. In moderation, provenance, and verification pipelines their output probability is read as a degree of trust, so its calibration matters as much as raw accuracy. We reframe deepfake detection as a calibrated, self-auditing trust instrument, the Calibrated Deepfake Trust Score (CDTS), and identify what governs its trustworthiness. Our central finding is a competence-calibration coupling: the calibration of the trust score degrades as the detector's discriminative competence falls. We establish it across 32 configurations (pooled Pearson r = -0.81), demonstrate it within a single dataset, reinforce it by inducing low competence directly, and replicate it on a fourth held-out dataset the detectors never trained on. It holds across three architecturally distinct detectors, two convolutional networks and a CLIP vision transformer (r = -0.88, -0.83, -0.86). The result is also deployable: a single calibrator frozen on in-domain data fails on exactly the low-competence generators the coupling flags (its error tracks competence at r = -0.98), and competence is estimable without labels, so a label-free monitor flags calibration risk on unseen generators and routing source-batches on a reference-free competence estimate lowers overall AURC and improves the low-to-mid coverage operating region relative to confidence-based routing. The same competence factor also drives calibration inequity across demographic subgroups (distinct from accuracy inequity) and explanation faithfulness. We therefore argue that detector trustworthiness is organized by competence as a shared driver, that competence is the right quantity to estimate and condition on, and that trust scoring must be competence-aware. We offer the CDTS wrapper as the mechanism, and report openly where the unification is tight and where it is architecture-specific.
