Search papers, labs, and topics across Lattice.
This paper introduces AIForge-Doc v2, a dataset of 3,066 GPT-Image-2 document forgeries with pixel-precise masks, and benchmarks the ability of humans, generic forensic tools (TruFor), document-specific tools (DocTamper), and GPT-Image-2 itself to detect these forgeries. The key finding is that humans perform at chance (50.1% accuracy) in distinguishing real from AI-edited receipts, and even GPT-Image-2 fails to reliably identify its own forgeries, achieving an AUC of only 0.59 as a self-judge. This highlights a significant vulnerability in document security as current detection methods struggle to identify even simple forgeries created by state-of-the-art AI models.
GPT-Image-2 can so seamlessly forge documents that neither humans nor the model itself can reliably tell the difference.
OpenAI's GPT-Image-2 has effectively erased the visual boundary between authentic and AI-edited document images: a single number on a receipt can be replaced in under a second for a few cents. We release AIForge-Doc v2, a paired dataset of 3,066 GPT-Image-2 document forgeries with pixel-precise masks in DocTamper-compatible format, and benchmark four lines of defence: human inspectors (N=120, n=365 pair-votes via the public 2AFC site CanUSpotAI.com), TruFor (generic forensic), DocTamper (qcf-568, document-specific), and the same GPT-Image-2 model as a zero-shot self-judge -- asked, to avoid the trivial"image is mostly real"reading, whether any region was generated or edited by an AI image model. Human 2AFC accuracy is 0.501, indistinguishable from chance: even side-by-side, inspectors cannot tell GPT-Image-2 receipt forgeries from authentic counterparts. The three computational judges sit only modestly above (TruFor 0.599, DocTamper 0.585, self-judge 0.532). The self-judge fails consistently, not by chance: across five prompt strategies and four policies for handling ambiguous responses, AUC never rises above 0.59. To rule out the possibility that the two forensic detectors are broken on our source domain rather than blind to AI inpainting, we calibrate each on a same-domain traditional-tampering set built for its training distribution: TruFor reaches AUC 0.962 on cross-camera splicing of our dataset, DocTamper reaches 0.852 on cross-document OCR-token splicing with two-pass JPEG re-encoding. Both retain near-published performance on traditional tampering; switching to GPT-Image-2 inpainting drops AUC by 0.27-0.36 (0.962->0.599 TruFor; 0.852->0.585 DocTamper), isolating a detection gap specific to GPT-Image-2 inpainting. We release the dataset, pipeline, four-judge protocol, and calibration sets.
