Search papers, labs, and topics across Lattice.
This paper introduces SciCodePile, a comprehensive 128GB corpus of scientific code derived from 37,737 public repositories, addressing the inadequacies of existing datasets for evaluating large language models (LLMs) in scientific code generation. The authors evaluate 15 LLMs on a curated executable benchmark of 200 tasks, revealing that current models struggle significantly, with the best achieving only 12.30% Pass@1 on executable tasks. Notably, continued pretraining on SciCodePile yields substantial improvements in performance, indicating its potential to enhance LLM capabilities in scientific coding tasks.
Current LLMs achieve only a 12.30% success rate in generating executable scientific code, highlighting a significant gap in their capabilities.
Large language models (LLMs) excel at general-purpose code generation, yet how well they handle scientific code remains an open question. Existing datasets and benchmarks are limited in scale, domain coverage, or executable verification, leaving the true gap between current LLMs and reliable scientific code generators inadequately assessed. To address these limitations, we present SciCodePile, the largest scientific code corpus to date, constructed from 37,737 public repositories and collectively comprising 128GB of code that spans multiple computational science disciplines. From this corpus, we further curate an executable benchmark of 200 tasks, each equipped with a sandboxed execution environment and an automated test harness for functional verification. We evaluate 15 LLMs from both open-source and closed-source families on three tasks: prefix-to-suffix completion, fill-in-the-middle infilling, and executable code generation. Results show that scientific code generation remains highly challenging: The best CodeBLEU reaches only 38.13 and 38.37 on the two completion tasks, while the strongest model achieves just 12.30\% Pass@1 on the executable benchmark, underscoring how far current models remain from reliable scientific code generation. To demonstrate the training utility of SciCodePile, we further show that continued pretraining on our corpus improves CodeBLEU by $\times$2.84 on scientific code completion, and instruction tuning on our data improves Pass@1 by $\times$4.79 on the executable benchmark. All code and data are available at https://huggingface.co/SciCodePile.
