Search papers, labs, and topics across Lattice.
This paper introduces a novel benchmark for evaluating LLMs in BIM-based design, along with a methodology for generating textual data from BIM models to create datasets for fine-tuning. They then fine-tune the Qwen model using their generated dataset and a specialized strategy, resulting in Qwen-BIM, a domain-specific LLM. Qwen-BIM demonstrates a 21% improvement in G-Eval score compared to the base LLM and achieves performance comparable to much larger general LLMs on BIM-related tasks.
A 14B parameter model, Qwen-BIM, rivals the performance of 671B parameter general LLMs in BIM-based design, thanks to a new domain-specific benchmark and dataset.
As the construction industry advances toward digital transformation, BIM (Building Information Modeling)-based design has become a key driver supporting intelligent construction. Despite Large Language Models (LLMs) have shown potential in promoting BIM-based design, the lack of specific datasets and LLM evaluation benchmarks has significantly hindered the performance of LLMs. Therefore, this paper addresses this gap by proposing: 1) an evaluation benchmark for BIM-based design together with corresponding quantitative indicators to evaluate the performance of LLMs, 2) a method for generating textual data from BIM and constructing corresponding BIM-derived datasets for LLM evaluation and fine-tuning, and 3) a fine-tuning strategy to adapt LLMs for BIM-based design. Results demonstrate that the proposed domain-specific benchmark effectively and comprehensively assesses LLM capabilities, highlighting that general LLMs are still incompetent for domain-specific tasks. Meanwhile, with the proposed benchmark and datasets, Qwen-BIM is developed and achieves a 21.0% average increase in G-Eval score compared to the base LLM model. Notably, with only 14B parameters, performance of Qwen-BIM is comparable to that of general LLMs with 671B parameters for BIM-based design tasks. Overall, this study develops the first domain-specific LLM for BIM-based design by introducing a comprehensive benchmark and high-quality dataset, which provide a solid foundation for developing BIM-related LLMs in various fields.
