Search papers, labs, and topics across Lattice.
This paper introduces REDACT, a comprehensive multilingual benchmark for detecting personally identifiable information (PII) that systematically controls for nine generation axes, resulting in 13,427 records and 324,078 entity annotations across 51 entity types and 25 languages. The evaluation of five PII detectors reveals that while LLM-based models exhibit robustness, traditional rule-based detectors struggle significantly with high-sensitivity data, achieving a recall of only 0.07 in critical categories. The findings highlight the importance of nuanced evaluation metrics, as aggregate F1 scores can obscure architecture-specific weaknesses in PII detection systems.
Traditional rule-based PII detectors falter on high-stakes data, revealing a critical vulnerability in current detection methodologies.
Benchmark infrastructure for personally identifiable information (PII) detection remains limited: existing corpora cover few entity types, use ad hoc generation conditions, and do not show which surface conditions cause detector failures. We present REDACT, a systematically controlled multilingual PII benchmark with 13,427 records, 324,078 entity annotations, 51 entity types, 4,127 surface-form patterns, and 25 languages across 9 scripts. A strength-2 covering-array sampler controls nine generation axes: domain, format, difficulty, length, density, code-switching, language, adjacency, and co-occurrence. Three entity-level metadata fields (disclosure status, disclosure form, and a GDPR-aligned sensitivity tier) enable stratified evaluation beyond aggregate or per-type F1. From the full benchmark, we evaluate five detectors (Presidio, GLiNER, the OpenAI Privacy Filter, GPT-4.1, and Claude Sonnet 4.6) on a locked, language-stratified sample of 1,000 records. Aggregate F1 masks an architecture-dependent failure structure: the rule-based detector performs poorly on the highest-stakes data, including HIGH-sensitivity categories (recall 0.07) and non-verbatim disclosure forms, while the LLM detectors remain more robust, with the HIGH tier as their strongest sensitivity slice. A three-model reference-free LLM-as-judge assessment corroborates that sensitivity-tier assignment is the task's hardest axis. We release the benchmark, schema, prompts, and stratified evaluation harness.
