Search papers, labs, and topics across Lattice.
This paper introduces TPS-CalcBench, a specialized benchmark designed to evaluate the analytical calculation competence of large language models (LLMs) in hypersonic thermal protection system engineering. By addressing the inadequacies of existing scientific benchmarks, TPS-CalcBench incorporates a dual-track evaluation system that assesses both result accuracy and reasoning quality, enabling the identification of critical failures in model outputs. The evaluation of 13 models reveals significant performance disparities and highlights the effectiveness of diagnostic interventions, establishing a robust framework for the safe deployment of LLMs in aerospace engineering contexts.
LLMs can produce numerically plausible yet physically invalid results in aerospace engineering, and TPS-CalcBench reveals the hidden dangers of relying solely on traditional benchmarks.
Deploying LLMs as reasoning assistants in safety-critical aerospace engineering requires stricter evaluation criteria than general scientific benchmarks. In hypersonic thermal protection system (TPS) design, inaccurate stagnation-point heat flux or boundary-layer calculations may cause catastrophic design margin violations. Models with numerically reasonable but physically invalid answers are more dangerous than those declining to respond. Current scientific benchmarks only test abstract math and basic physics, evaluate final answers solely, ignore engineering reasoning processes, and cannot detect such critical failures. We propose TPS-CalcBench, the first diagnostic benchmark for closed-form analytical calculations in hypersonic aerodynamics and high-temperature gas dynamics that experienced TPS engineers conduct without simulations. Our contributions include domain-oriented task taxonomy with 4 difficulty levels and 8 categories from Anderson's textbook, dual-track evaluation measuring result accuracy and reasoning quality via an 8-dimension rubric and calibrated judge with human audit to identify right answer wrong reasoning issues, human-AI data pipeline producing 420 high-confidence core items and 810 noise-controlled pre-gating items from 4560 raw data, noise-sensitivity analysis measuring data quality impacts on model ranking, and three diagnostic intervention methods: DFA-TPS fine-tuning, RAG-EQ retrieval grounding and PA-CoT process-aware prompting. Tests on 13 models from 7 groups show wide performance differences (KPI 12.6-87.9), hidden formula selection defects, data-driven rank changes and effective intervention improvements, establishing a complete diagnose-evaluate-intervene framework for safety-critical engineering LLM deployment assessment.
