Search papers, labs, and topics across Lattice.
This paper introduces ConfBench, a novel calibration-specific benchmark designed to evaluate the confidence scores of vision-language models (VLMs) in the context of key information extraction (KIE) from documents. By applying 20 controlled degradation pipelines to a diverse set of documents, the authors generated 1,346 variants and over 70,000 entity-level evaluations, facilitating a comprehensive assessment of model confidence across varying accuracy levels. The findings reveal significant insights into the relationship between model capability and confidence quality, highlighting the need for tailored calibration methods to improve trustworthiness in intelligent document processing applications.
Calibration quality of VLMs varies dramatically, with some models exhibiting near-perfect confidence estimates while others are severely overconfident, impacting automation decisions in document processing.
Intelligent document processing (IDP) with vision-language models (VLMs) hinges on confidence scores trustworthy enough to route extractions between automation and human review. Existing document benchmarks are dominated by clean, high-quality samples, leaving low accuracy regions too sparse for calibration assessment. We introduce ConfBench, the first calibration-specific benchmark for key information extraction (KIE), built by applying 20 controlled degradation pipelines to a diverse document set, yielding 1,346 variants and 70K+ entity-level evaluations spanning the full accuracy spectrum. We evaluate four proprietary and three open-weight VLMs under verbalized and log-probability confidence estimation methods across three input modalities, and find: (i) OCR+Image modality results in more accurate confidence estimates; (ii) model capability is the dominant factor: within the Claude family confidence quality scales monotonically with capability, while across families parameter count is a poor predictor; (iii) calibration quality varies widely across models, from near-perfect to severely overconfident, and per-model post-hoc correction rescales these absolute confidence values for threshold-based routing without altering ranking-based operational metrics; and (iv) log-probability with first-token aggregation consistently outperforms mean-token and margin aggregations. We also introduce ECARB, a review-budget metric translating discriminative gains into operational savings. We release ConfBench publicly to enable systematic study of confidence estimators and calibration methods for trustworthy IDP application deployment.
