Search papers, labs, and topics across Lattice.
This paper introduces NuclearQAv2, a structured benchmark designed to evaluate the competence of large language models (LLMs) specifically in the domain of nuclear engineering. The benchmark includes approximately 1,240 question-answer pairs across boolean, numeric, and verbal categories, generated through a hybrid approach that combines expert input, existing datasets, and LLM-assisted techniques. Evaluation results reveal that while LLMs excel at factual questions, they struggle significantly with quantitative reasoning and conceptual understanding, underscoring the need for comprehensive evaluation frameworks in technical fields.
LLMs excel at factual recall but falter on quantitative reasoning and conceptual tasks, revealing critical gaps in their domain-specific capabilities.
Large language models (LLMs) have demonstrated strong performance across a wide range of tasks, but ensuring their reliability in highly technical domains remains a significant challenge. In nuclear engineering, problem solving often requires not only factual knowledge but also quantitative reasoning and conceptual understanding. To address the need for systematic evaluation in this domain, we introduce NuclearQAv2, a benchmark for assessing LLMs on nuclear engineering knowledge. The benchmark comprises approximately 1,240 question-answer pairs spanning three categories: boolean, numeric, and verbal. NuclearQAv2 is constructed using a hybrid pipeline that combines expert-authored questions, existing datasets, and LLM-assisted generation from domain-specific technical corpora. By leveraging structured prompting for both automated question generation and response evaluation, the proposed framework enables scalable benchmark construction and evaluation. We evaluate a diverse set of LLMs using NuclearQAv2 and observe substantial performance differences across task types. While the models generally perform well on factual questions, quantitative reasoning and conceptual understanding remain considerably more challenging. These results highlight the importance of multi-faceted evaluation frameworks and establish NuclearQAv2 as a scalable benchmark for assessing LLM capabilities in technical domains.
