Search papers, labs, and topics across Lattice.
The paper introduces Double Triangle Annotation, a two-layer human-in-the-loop framework that leverages cross-model consensus between multimodal LLMs to automate structured information extraction from historical documents. By requiring agreement between independent models and escalating disagreements to human annotators and domain experts, the framework achieves a Word Error Rate of 0.003 on the Guides Rosenwald dataset. The approach auto-accepts over 85% of fields, demonstrating its scalability and efficiency in generating high-precision annotations.
Achieve near-perfect structured information extraction from historical documents with a surprisingly simple framework that auto-accepts the vast majority of labels via cross-model consensus.
Evaluating structured-information extraction from historical documents at scale requires high-precision ground-truth annotations, yet traditional manual labeling is expensive and fully automated pipelines built on large language models are prone to hallucination. We propose Double Triangle Annotation, a two-layer human-in-the-loop framework that leverages cross-model consensus to automate the majority of annotation work while ensuring high-precision outputs. In the first layer, two architecturally independent Multimodal Large Language Models annotate each document in parallel; when they agree, the label is auto-accepted, and disagreements are routed to a human jury. A second layer cross-checks two such systems against each other, escalating residual conflicts to a domain expert. The framework rests on a single assumption -- error independence between models -- requires no distributional priors or task-specific calibration, and becomes more autonomous as model capability improves. On the Guides Rosenwald, a corpus of French medical directories spanning 1887-1906, the framework achieves a final Word Error Rate of 0.003. Applied at scale, model consensus auto-accepts over 85% of 13,595 fields. We release the resulting benchmark -- the first structured-extraction ground truth for the Rosenwald Guides -- to support future work on historical document processing.
