Search papers, labs, and topics across Lattice.
The paper introduces SADU, a new benchmark dataset for evaluating Vision-Language Models (VLMs) on their ability to understand software architecture diagrams, encompassing behavioral, structural, and ER diagrams. The benchmark includes 154 diagrams and 2,431 question-answer tasks focusing on counting and retrieval reasoning. Evaluation of 11 state-of-the-art VLMs reveals that understanding software architecture diagrams remains a significant challenge, with the best model achieving only 70.18% accuracy, exposing weaknesses in diagram reasoning and visual relation grounding.
Despite advances in VLMs, understanding software architecture diagrams remains surprisingly difficult, with even top models struggling to surpass 70% accuracy on a new benchmark designed to test diagram reasoning and visual grounding.
Software architecture diagrams are important design artifacts for communicating system structure, behavior, and data organization throughout the software development lifecycle. Although recent progress in large language models has substantially advanced code-centric software engineering tasks such as code generation, testing, and maintenance, the ability of modern vision-language models (VLMs) to understand software architecture diagrams remains underexplored. To address this gap, we present SADU, a benchmark for Software Architecture Diagram Understanding that evaluates VLMs on architecture diagrams as structured software engineering artifacts rather than generic images. SADU contains 154 carefully curated diagrams spanning behavioral, structural, and ER diagrams, paired with structured annotations and 2,431 question-answer tasks covering counting and retrieval reasoning. We evaluate 11 state-of-the-art VLMs from the Gemini, Claude, GPT, and Qwen families. Our results show that software architecture diagram understanding remains challenging for current models: the best-performing model gemini-3-flash-preview achieves only 70.18\% accuracy, while gpt-4o-mini only achieves 17.77\% accuracy. The results further reveal the weaknesses in diagram reasoning and visual relation grounding, highlighting a gap between current VLMs and the needs of design-stage software engineering. SADU provides a foundation for future research on diagram-aware AI systems and more faithful AI-assisted software engineering workflows.
