Search papers, labs, and topics across Lattice.
This paper introduces LeakageBench, a benchmark designed to evaluate the risk of document-level leakage of personally identifiable information (PII) in document images, addressing the limitations of existing text-centric benchmarks. By analyzing 500 document images with nearly 12,000 PII annotations, the study assesses the performance of various OCR pipelines and vision-language models in detecting PII, revealing that while tool assistance can enhance localization, significant leakage risks persist. The findings highlight that even with improved detection methods, critical page-level leakage remains alarmingly high, underscoring the need for better redaction strategies in document images.
Despite advancements in OCR and vision-language models, critical page-level leakage of PII remains dangerously high, reaching 0.968 even with improved detection methods.
Real-world personally identifiable information (PII) redaction often operates on document images---scans, screenshots, and PDF renderings---where OCR errors, layout structure, and visual noise determine whether sensitive information is actually removed. Existing PII benchmarks are mostly text-centric and do not measure document-level redaction risk: a page remains unsafe if even one identifier is missed. We introduce LeakageBench, a challenge set of 500 document images with 11,954 GDPR-aligned PII annotations spanning direct identifiers, linkage keys, and contextual re-identification surfaces. We evaluate generic OCR pipelines, commercial and task-adapted OCR-dependent detectors, and OCR-free vision-language models using entity-level F1, group-wise leakage, and document-level leakage metrics. Code Interpreter raises GPT-5.5 localization F1 from 0.090 to 0.249, but critical page-level leakage remains 0.968. These results show that stronger detection and tool assistance improve localization without making most pages safe for release. LeakageBench provides a diagnostic benchmark for high-recall, spatially grounded PII redaction in document images.
