Search papers, labs, and topics across Lattice.
This paper introduces a document-understanding system utilizing a Mixture-of-Experts VLM, fine-tuned on a combination of in-house production data and open-domain documents curated through a Difficulty-Aware pipeline. The system achieves significant cost reductions, lowering expected costs by over 80% compared to human annotation and more than 50% against the best competing open-source model, while outperforming deployable baselines up to ten times larger. These advancements highlight the potential for economically viable, high-quality document processing in regulated industries where traditional methods fall short.
A novel Mixture-of-Experts VLM slashes document processing costs by over 80%, outperforming human annotation and existing models.
Extracting structured fields from hundreds of millions of documents annually remains costly in regulated industries: bespoke OCR cascades cover only a fraction of workflows, privacy rules preclude external models, and existing open-source VLMs that clear quality thresholds cost more to serve than human annotation. We present a deployed document-understanding system built on a Mixture-of-Experts VLM (35B total, 3B active), fine-tuned on in-house production data mixed with open-domain documents curated by a Difficulty-Aware pipeline for layout diversity, fact-extractability, and cross-model consistency. Fitting on a single H100 and serving heterogeneous workflows via prompting, the model leads all deployable (non-reasoning) baselines up to an order of magnitude larger. A quality-adjusted cost analysis, with confirmation and correction costs calibrated from production telemetry, shows it reduces expected costs by over 80% against the human baseline and by more than 50% against the best competing open-source model, while larger baselines remain economically unviable.
