Search papers, labs, and topics across Lattice.
This paper introduces ExtractConf, a novel confidence estimation engine designed for large language model (LLM)-based document field extraction, addressing the critical need for reliable trustworthiness assessments in high-stakes applications. By leveraging two distinct extraction approaches鈥攆ield-guided and document-guided鈥攖he system effectively captures and fuses the inherent uncertainties and disagreements between the two methods, leading to improved accuracy and reduced risk of erroneous extractions. The results demonstrate that ExtractConf achieves a ROC AUC of 0.928 on the DocILE dataset, significantly enhancing selective prediction risk management and enabling a robust human-in-the-loop workflow.
ExtractConf reduces selective prediction risk by 70% over traditional log-probability methods, transforming how we assess trustworthiness in LLM-driven document processing.
In high-stakes document processing pipelines, including financial reconciliation, compliance verification, and procurement automation, an LLM extraction that is silently wrong is more dangerous than one that is visibly absent. The central challenge is not extraction accuracy alone but reliable confidence estimation: knowing, field by field, whether an extraction can be trusted for automation or deferred to human review. Token-level log-probabilities, verbalized confidence, and multi-sample self-consistency all collapse toward all-positive behaviour at practical thresholds, offering no reliable separation between trustworthy and untrustworthy extractions. We present ExtractConf, a cross-domain, field-agnostic confidence engine that grounds confidence estimation in two structurally different readings of the same document. A field-guided Hunter call extracts each field under schema-slot completion pressure; a document-guided Mapper call scans holistically and surfaces values grounded in document content. This asymmetry yields different failure modes: Hunter hallucinates values for absent fields, while Mapper misses visually non-salient ones. Their disagreement is independently informative. ExtractConf fuses cross-call disagreement, LLM-internal uncertainty, OCR, image quality, and spatial layout into a classifier requiring no domain-specific rules or retraining. On DocILE (55-field invoices, 26% failure rate), it achieves 0.928 ROC AUC and reduces selective prediction risk by 70% over logprob-mean. At 80% coverage, accuracy reaches 99.1%, enabling a practical human-in-the-loop workflow. Zero-shot transfer to CORD receipts achieves 0.858 AUC; lightweight Lasso recalibration reduces ECE by 89% and Brier by 43%, confirming the signals generalise across document domains.
