Search papers, labs, and topics across Lattice.
The authors evaluate LLM automated document auditing by injecting 450 controlled typographical, semantic, and absurd contaminants across 150 academic papers and testing Gemini 3.0 Pro across scaling batch sizes. Detection collapsed from 60% on small batches to just 2.8% on large batches, accompanied by a deceptive failure mode where the model hallucinated confident, stylistically matched errors rather than abstaining. Crucially, realistic semantic reversals were missed far more frequently (50% recovery) than blatant absurdities (75%), demonstrating that LLM auditors are least effective at catching the high-stakes corruptions most likely to appear in the wild.
Scaling LLM document auditing to large batches causes detection recall to crash from 60% to 2.8%, triggering confident hallucinations of nonexistent errors rather than graceful abstention.
Large language models are increasingly proposed as automated auditors of document quality, yet their reliability as detectors of planted errors is poorly characterised. We construct a contaminated corpus of 150 academic papers spanning supply chain management and medical research, injecting 450 known contaminants of three types: typographical corruption, semantic reversal, and absurd out-of-context insertion. We then evaluate Google Gemini 3.0 Pro's ability to recover a 180-contaminant answer-key subset across 60 documents under three prompting regimes of increasing scale: single document, small batch, and large batch. Detection holds at small scale and then collapses: 50% recovery on single documents, 60% on small batches, and 2.8% on large batches. The failure mode at scale is not abstention but fabrication. Rather than reporting incomplete processing, the model produced confident findings including invented contaminants of its own, absurdities such as"telepathic squirrel"and"quantum-powered toaster"that mimic the style of the planted material but do not appear in any document. Detection also varies by contamination type: absurd insertions were recovered at 75% in completed evaluations, while semantic reversals and typographical corruptions were each recovered at only 50%. The corruptions most likely to occur in the wild, plausible ones, are the ones most often missed. We conclude that LLM document auditing degrades not gracefully but deceptively, and outline the harness such systems require: bounded batch sizes, direct content injection, and mechanical verification of every reported finding against source text.
