Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
To evaluate autonomous agents on complex industrial workflows requiring spatial and physical reasoning, the authors construct EngiWorld, a benchmark spanning 1,301 expert-curated tasks across six core engineering domains (CAD, CAE, CAM, BIM, EDA, and 3D visualization) and 26 professional software suites. The benchmark implements an artifact-centric evaluation pipeline featuring a programmatic verifier suite that continuously scores intermediate and final outputs against geometric validity, physical feasibility, and strict specification constraints. Across seven evaluated frontier models, even the top-performing system achieved an EngiScore of only 44.3, with task completion collapsing to an abysmal 3.6% on multi-software workflows.
Today's frontier agents virtually collapse in professional engineering environments, solving just 3.6% of multi-software workflows when held to real-world physical and geometric constraints.
Autonomous agents have made rapid progress in general-purpose computer use, but reliable automation of professional industrial engineering remains out of reach, as engineering workflows demand reasoning over geometric and physical constraints and dependencies preserved across software and design stages. We present EngiWorld, the first benchmark structured around the complete design loop: 1,301 expert-curated tasks spanning 6 engineering domains (CAD, CAE, CAM, BIM, EDA, and 3D visualization) and 26 professional software platforms, with both GUI and CLI interfaces and 6 task types ranging from software-selection to open-ended tasks. We further introduce an artifact-centric evaluation methodology built on a unified domain-verifier suite, which programmatically checks the geometric validity, physical feasibility, and rule compliance of final and intermediate artifacts, and scores quantitative design tasks continuously by specification attainment rather than binary success. Evaluation of seven frontier models reveals a substantial capability gap: the strongest model achieves an EngiScore of only 44.3, and just 3.6% of multi-software attempts succeed. EngiWorld provides the first rigorous foundation for measuring progress toward agents that operate professional engineering software end to end.
Tsinghua AI