Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper details the construction of a hybrid HPC/AI software stack using Nix, addressing the challenges of reproducibility in high-performance computing environments where traditional methods fall short. By leveraging Nix's consistent package management and environment isolation, the authors successfully integrated C/C++ and Python dependencies into a single declarative specification, facilitating both local development and remote deployment on supercomputers. Key findings highlight the advantages of Nix over other tools like Spack and Guix, particularly in resolving dependency issues and ensuring reproducibility without requiring root access.
Nix transforms HPC software management by unifying diverse dependencies into a single, reproducible framework that thrives in restrictive environments.
Reproducibility in HPC remains difficult under the constraints of production supercomputers: no root access, limited internet, and software stacks that increasingly span C/C++, Fortran, Python, MPI, and GPU runtimes. Traditional approaches based on environment modules and Conda require manual intervention to locate dependencies, leak system libraries into builds, and fail to compose across projects. Containers help with deployment but do not by themselves guarantee reproducibility. We report on our experience building a hybrid HPC/AI software stack with Nix, covering local development on a workstation without root and remote deployment as an Apptainer image on a production cluster. Nix's consistent package layout, full environment isolation, and flake-based composition resolve the dependency discovery, leakage, and composition problems we encountered, while unifying C/C++ and Python management under a single declarative specification that also generates the deployment container. We discuss trade-offs against Spack and Guix, the development-versus-production split addressed via CMake presets, and current gaps in ML package coverage in Nixpkgs.