Search papers, labs, and topics across Lattice.
The paper introduces GDP.pdf, a benchmark designed to evaluate grounded multimodal reasoning capabilities specifically within professional PDF documents. By focusing on realistic question-document pairs authored by professionals across ten fields, the benchmark effectively identifies the limitations of current multimodal models, revealing that even the best-performing model only passed 15% of the items. The findings highlight critical recurring error patterns, such as misaligned tables and misread charts, underscoring the need for improved document AI systems that can handle complex, real-world queries.
Despite advances in multimodal models, even the best can only answer 15% of realistic questions from professional PDFs, exposing significant gaps in current capabilities.
A large share of day-to-day work in professional domains happens inside PDF files: benefits packets, leases, datasheets, clinical guidelines, construction plans. Benchmarks for document AI have generally measured the required capabilities in isolation: OCR, layout analysis, chart reasoning, table QA, document VQA. A high score on any one of them does not necessarily reveal whether a model can answer a realistic question that someone in the field would actually ask about a specific PDF. GDP.pdf is a benchmark built to measure this directly. It consists of question-document pairs authored by working professionals in ten fields, and a candidate question was kept only when at least two frontier multimodal models failed it in a way that mattered: a wrong answer, missed decisive evidence, or a fabricated claim, rather than a superficial difference such as style. Each item comes with a rubric of atomic criteria, so we can report a graded rubric score as well as a strict task-level pass rate, and each item is tagged against a taxonomy of eleven capabilities in three tiers, spanning text extraction and grounding, table and chart comprehension, cross-referencing, spatial reasoning, and abstention on unsupported queries. We evaluated seven frontier models on the 100-item benchmark. The best model passed only 15% of the items and the worst passed 1%. Most errors trace back to a small set of recurring loss patterns: misaligned tables, misread charts, skipped footnotes and exclusions, miscounted floor-plan symbols, scan noise, and amendments that supersede earlier text. The full 100-item benchmark is publicly available at https://huggingface.co/datasets/surgeai/GDP.pdf
