Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The paper introduces Layout Error Detection (LED), a new benchmark for evaluating structural errors in Document Layout Analysis (DLA) predictions that goes beyond surface-level accuracy. LED defines eight standardized error types and provides quantitative rules and injection algorithms for realistic error simulation, resulting in the LED-Dataset. Experiments with state-of-the-art multimodal models demonstrate that LED enables fine-grained assessment of structural understanding and reveals weaknesses in existing models.
Current document layout analysis metrics miss critical structural errors like region merging and splitting, so this benchmark offers a much-needed, more comprehensive evaluation.
Recent advances in Large Language Models (LLMs) and Large Multimodal Models (LMMs) have improved Document Layout Analysis (DLA), yet structural errors such as region merging, splitting, and omission remain persistent. Conventional overlap-based metrics (e.g., IoU, mAP) fail to capture such logical inconsistencies. To overcome this limitation, we propose Layout Error Detection (LED), a benchmark that evaluates structural reasoning in DLA predictions beyond surface-level accuracy. LED defines eight standardized error types (Missing, Hallucination, Size Error, Split, Merge, Overlap, Duplicate, and Misclassification) and provides quantitative rules and injection algorithms for realistic error simulation. Using these definitions, we construct LED-Dataset and design three evaluation tasks: document-level error detection, document-level error-type classification, and element-level error-type classification. Experiments with state-of-the-art multimodal models show that LED enables fine-grained and interpretable assessment of structural understanding, revealing clear weaknesses across modalities and architectures. Overall, LED establishes a unified and explainable benchmark for diagnosing the structural robustness and reasoning capability of document understanding models.