Search papers, labs, and topics across Lattice.
This paper introduces EngVQA, a novel multimodal benchmark designed to evaluate engineering reasoning in Vision-Language Models (VLMs) across five subjects with 696 problems. The authors implement an 8-stage automatic evaluation framework that assesses VLM-generated solutions at each reasoning step, revealing significant limitations in current models' engineering reasoning capabilities. Human evaluations corroborate the automated framework's findings, underscoring the necessity for process-oriented assessments in multimodal reasoning tasks.
Current Vision-Language Models struggle with engineering reasoning, revealing critical gaps in their ability to interpret technical diagrams and apply physical principles.
Vision-Language Models (VLMs) demonstrate strong performance on general multimodal reasoning benchmarks, yet their ability to perform engineering reasoning remains largely unexplored. Unlike general visual question answering, engineering problem solving requires interpreting technical diagrams, selecting governing physical principles, and maintaining physically consistent multi-step reasoning. These capabilities are increasingly important for AI systems used in engineering education, scientific assistance, and technical decision-making, where reasoning failures may produce physically invalid yet superficially plausible solutions. Existing benchmarks primarily evaluate final answers and provide limited assessment of intermediate reasoning processes. We introduce EngVQA, a multimodal benchmark for evaluating engineering reasoning across 5 engineering subjects containing 696 problems. We introduce an 8-stage automatic evaluation framework for assessing VLM-generated solutions. The framework independently evaluates each stage of the solution, enabling fine-grained analysis of reasoning failures. We benchmark multiple state-of-the-art open and closed source VLMs on our evaluation framework and demonstrate substantial limitations in current engineering reasoning capabilities. Human evaluation shows strong agreement with our automated framework, achieving a Pearson correlation of 0.975 and a mean absolute error of 0.67 on a 10-point grading scale. Our results highlight the importance of process-oriented evaluation for reliable assessment of multimodal engineering reasoning systems.
