Search papers, labs, and topics across Lattice.
The paper introduces Text2Arch, a new dataset of scientific architecture diagrams paired with natural language descriptions and DOT code representations, designed to train models for generating diagrams from text. They fine-tuned small language models on this dataset and compared their performance to GPT-4o and DiagramAgent, finding that the fine-tuned models significantly outperform existing baselines and achieve comparable results to GPT-4o's in-context learning. The release of the dataset, code, and models aims to foster further research in AI-driven diagram generation.
Forget struggling with ambiguous text descriptions – now you can automatically generate high-fidelity scientific architecture diagrams from natural language, thanks to a new dataset that rivals GPT-4o's diagramming abilities.
Communicating complex system designs or scientific processes through text alone is inefficient and prone to ambiguity. A system that automatically generates scientific architecture diagrams from text with high semantic fidelity can be useful in multiple applications like enterprise architecture visualization, AI-driven software design, and educational content creation. Hence, in this paper, we focus on leveraging language models to perform semantic understanding of the input text description to generate intermediate code that can be processed to generate high-fidelity architecture diagrams. Unfortunately, no clean large-scale open-access dataset exists, implying lack of any effective open models for this task. Hence, we contribute a comprehensive dataset, \system, comprising scientific architecture images, their corresponding textual descriptions, and associated DOT code representations. Leveraging this resource, we fine-tune a suite of small language models, and also perform in-context learning using GPT-4o. Through extensive experimentation, we show that \system{} models significantly outperform existing baseline models like DiagramAgent and perform at par with in-context learning-based generations from GPT-4o. We make the code, data and models publicly available.
Microsoft Research