Search papers, labs, and topics across Lattice.
This study evaluates eight open-source pretrained Vision-Language Models (VLMs) on Document Visual Question Answering (DocVQA) across three document domains: industrial documents, infographics, and presentation slides. The research reveals that while larger VLMs perform well in zero-shot scenarios for structured layouts, they struggle with the complexity of visually rich documents, highlighting visual understanding as a key bottleneck. Notably, smaller models benefit significantly from supervised finetuning, achieving rapid adaptation to new domains and sometimes outperforming fully supervised models with limited target domain samples.
Visual understanding, not knowledge, is the critical barrier in Document Visual Question Answering, with smaller models showing surprising adaptability through targeted finetuning.
Document Visual Question Answering (DocVQA) presents a complex multimodal challenge, requiring models to exploit visual, textual, and layout information from documents. Although Vision-Language Models (VLMs) have shown remarkable performance in text-vision tasks, their robustness and transferability to different document domains remains underexplored. In this study, we present a comprehensive evaluation of 8 open-source pretrained VLMs on DocVQA in three different document domains: industrial documents of varying type, infographics, and presentation slides. We systematically assess model performance under zero-shot evaluations, fully supervised finetuning with inter- and intra-dataset evaluations, and few-shot learning evaluations of knowledge transfer between domains. Our findings demonstrate that while large pretrained VLMs possess strong zero-shot baselines for structured layouts, their performance strongly decreases on visually complex layouts of infographics and slides. Although parameter scaling is a dominant factor on performance, supervised finetuning yields higher relative gains in smaller architectures. Furthermore, our cross-domain and few-shot experiments show that visual understanding is the main bottleneck for DocVQA, not a lack of knowledge from the VLMs. Using 50 target domain samples, the models finetuned in DocVQA with datasets of different domains rapidly adapt to the target domain documents, even surpassing their fully supervised counterparts in some cases.
