Search papers, labs, and topics across Lattice.
This paper introduces ExtractBench, a comprehensive benchmark designed for evaluating schema-guided document extraction in enterprise workflows, focusing on metrics such as value accuracy, record completeness, grounding, and cost. The benchmark comprises 4,869 pages from 370 documents across various business domains, facilitating a robust evaluation of extraction performance. Key findings reveal that while commercial VLMs excel with short documents, they struggle with longer ones, whereas the LlamaExtract Agentic Plus achieves top performance across metrics with comparable accuracy to coding agents at significantly lower costs.
LlamaExtract Agentic Plus outperforms commercial VLMs in schema-guided extraction, achieving high accuracy at a fraction of the cost.
Enterprise workflows increasingly rely on agents for schema-guided extraction: given a document and a user-defined schema, the agent faithfully follows the schema to produce the correct output with source evidence as grounding metadata. We present ExtractBench, a benchmark for schema-guided extraction and, to our knowledge, the first to score value accuracy, record completeness at scale, grounding, and measured cost together. The evaluation system contains 4,869 pages across 370 enterprise documents, 8 business domains, and 67 document types, with clear tags differentiating their challenge scenarios. The scalable schema and ground-truth curation pipeline combines independent-system agreement for real documents, known values for synthetic lists, and human verification for forms. We report order-insensitive value F1 for value accuracy, plus two grounding metrics for source traceability: word- and page-level F1. Commercial VLMs perform well on short documents but often truncate record lists on long ones, while coding agents retain higher accuracy at much higher cost. LlamaExtract Agentic Plus ranks first on all three metrics, with accuracy comparable to coding agents at a fraction of the cost. Dataset and evaluation code are available on https://huggingface.co/datasets/llamaindex/ExtractBench{HuggingFace} and https://github.com/run-llama/ExtractBench{GitHub}.
