Search papers, labs, and topics across Lattice.
This paper explores the integration of large language model (LLM) coding assistants with the FABRIC testbed to enhance computational reproducibility in scientific research. By reproducing three diverse case studies, the authors demonstrate that the AI assistant significantly reduces the effort required for experiment reproduction, achieving a 4-6 times reduction in workload. However, the study also reveals limitations in the AI's ability to handle complex analysis stages, underscoring the need for human intervention in establishing execution order and data dependencies.
AI-assisted workflows can cut down experiment reproduction efforts by up to six times, but struggle with complex analysis tasks requiring human oversight.
Computational reproducibility remains difficult despite being central to scientific research. In this paper, we show how the international FABRIC testbed, combined with large language model (LLM) coding assistants through LoomAI, can simplify reproducing published experiments across multiple domains. We reproduced three case studies on FABRIC, covering BBR-family congestion-control evaluations, LAMMPS molecular dynamics scaling benchmarks on a CPU-only MPI cluster, and stress protein homeostasis genomics pipelines. Rather than focusing only on matching numerical outputs, we evaluate whether the reproduced experiments support the same scientific conclusions as the original studies. The AI assistant was effective in setting up the environment, adapting code, and debugging, but struggled with the analysis stages that lacked clearly defined workflows, which required human guidance to establish execution order and data dependencies. Across the case studies, the AI-assisted workflow reduced reproduction effort by roughly 4--6 times. We conclude with practical recommendations for improving AI-assisted reproducibility on research testbeds.
