Search papers, labs, and topics across Lattice.
Atompack is a novel storage format and distribution layer tailored for the unique demands of atomistic machine learning datasets, focusing on efficient handling of read-heavy workloads. By optimizing the process of appending records and utilizing a memory-mapped read path, Atompack achieves a remarkable 96x speedup over existing solutions like ASE LMDB for shuffled training-style reads, while also reducing artifact size by approximately 79%. This advancement not only enhances the efficiency of training pipelines but also ensures that datasets remain compact and easily distributable as scientific artifacts.
Atompack achieves a staggering 96x improvement in read performance for atomistic ML datasets, revolutionizing how we handle training data efficiency.
Atomistic machine learning datasets are increasingly used for training: large immutable snapshots are read repeatedly, shuffled across epochs, staged across clusters' storage systems, and republished as reusable scientific artifacts. This workload differs from interactive scientific curation, where mutable records and ad hoc inspection are often more important than random indexed throughput. We present Atompack, an append-oriented storage format and distribution layer designed around a simple workload: training pipelines usually consume complete molecular records, while the order of records is randomized by the learning algorithm. Atompack appends records efficiently during dataset construction, then commits an immutable index and serves records through a memory-mapped read path optimized for training. We compare Atompack with HDF5, LMDB, and ASE baselines representing array stores, key-value records, serialized records, and object-oriented databases. The benchmarks measure sequential reads, shuffled reads, shared-filesystem behavior, write throughput, and artifact size. On a representative 64-atom workload, Atompack is 96x faster than ASE LMDB on shuffled training-style reads while producing artifacts about 79\% smaller. The results indicate that serving complete molecule records, rather than field chunks or reconstructed objects, improves shuffled training throughput while keeping artifacts compact enough for public distribution.
