Search papers, labs, and topics across Lattice.
The paper introduces CelloAI Benchmarks, a new suite of benchmarks designed to evaluate LLM performance in HEP/HPC software development, addressing the limitations of existing benchmarks that don't capture the specific constraints of these domains. The benchmarks consist of three tracks: code documentation (Doxygen), code generation (GPU kernels), and graphical data analysis (vision-enabled LLMs). Results from these benchmarks enable repeatable, automated scoring and fair comparisons of LLMs for scientific coding assistance, highlighting domain-relevant failure modes.
LLMs struggle with the nuances of scientific code, so CelloAI offers a practical benchmark to measure and improve their reliability in HEP/HPC software development.
Large Language Models (LLM) are increasingly used for software development, yet existing benchmarks for LLM-based coding assistance do not reflect the constraints of High Energy Physics (HEP) and High Performance Computing (HPC) software. Code correctness must respect science constraints and changes must integrate into large, performance-critical codebases with complex dependencies and build systems. The primary contribution of this paper is the development of practical, repeatable benchmarks that quantify LLM performance on HEP/HPC-relevant tasks. We introduce three evaluation tracks -- code documentation benchmarks measure the ability of an LLM to generate Doxygen-style comments, code generation benchmarks evaluate end-to-end usability on representative GPU kernels, and graphical data analysis benchmarks evaluate vision-enabled LLMs. These benchmarks provide a unified framework for measuring progress in scientific coding assistance across documentation quality, code generation robustness, and multimodal validation analysis. By emphasizing repeatability, automated scoring, and domain-relevant failure modes, the suite enables fair comparisons of models and settings while supporting future work on methods that improve reliability for HEP/HPC software development.
