Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces BIM-Edit, a benchmark designed to evaluate large language models (LLMs) on their ability to edit Building Information Models (BIM) in the Industry Foundation Classes (IFC) format. The benchmark consists of 324 editing tasks that assess models on geometric accuracy, semantic validity, and topological consistency, reflecting the complexities of real-world engineering design. Results reveal that even the best-performing LLM only achieves an average score of 49.5%, highlighting a significant gap in LLM capabilities for structured engineering tasks.
LLMs struggle with BIM editing, achieving less than 50% accuracy on critical engineering tasks, revealing a stark challenge for AI in design workflows.
Large language models (LLMs) are increasingly applied to computer-aided design (CAD) to generate design artifacts from textual instructions. In engineering practice, this requires more than creating new geometry, models must also understand existing scenes, edit them correctly, and preserve semantics and relations. However, many CAD benchmarks focus on creating new models rather than editing existing ones, and mostly evaluate geometric correctness. We introduce BIM-Edit, a benchmark for evaluating LLMs on natural-language editing of Building Information Models (BIM) represented in the Industry Foundation Classes (IFC) format. BIM provides a challenging testbed because building models encode geometry together with semantic and relational structure. BIM-Edit contains 324 editing tasks spanning 11 realistic building models and 36 synthetic scenes. Tasks are expressed using three instruction categories - direct, spatial, and topological - covering both explicit and scene-grounded edits. We evaluate outputs along three dimensions: geometric accuracy, semantic validity, and topological consistency. Across evaluated LLMs, the best-performing model achieves only 49.5% average score across the three metrics, and no model fully solves more than 3.4% of tasks. These results demonstrate a substantial gap between current LLM capabilities and the requirements of structured engineering design workflows.