Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
ChemReporter is a modular framework designed to curate and export large-scale chemical datasets for machine learning interatomic potential (MLIP) training, addressing the challenges of dataset quality and diversity. It operates through three stages: processing raw datasets into a structured repository, querying this repository with flexible selection criteria, and exporting the curated data in a format compatible with MLIP frameworks. The key finding is that ChemReporter can efficiently handle datasets larger than available memory, enabling the processing of billion-structure datasets on standard computing resources while ensuring reproducibility and traceability of data sources.
ChemReporter can streamline the curation of massive chemical datasets, enabling MLIP training on billion-structure datasets without memory constraints.
Training set quality and diversity are key determinants of the reliability of machine learning interatomic potentials (MLIPs), yet using massive datasets in full is often impractical and redundant, making intelligent data selection essential. A major bottleneck, however, is the lack of infrastructure for uniformly accessing, curating, and subsampling heterogeneous large-scale chemical datasets, which differ widely in structure, metadata, and file format. We address this gap with ChemReporter, a modular, method-agnostic framework that converts arbitrary molecular and materials datasets into a unified, queryable representation and exports the results directly into MLIP-ready training data. ChemReporter operates in three decoupled stages: processing, which parses raw datasets into a partitioned Apache Parquet repository enriched with structural, physical, and chemical metadata; querying, which filters and samples this repository via a CLI or Python API using arbitrary selection criteria, from simple physical constraints to custom, user-defined strategies; and exporting, which streams the selected subset into an HDF5 file ready for direct use in modern MLIP training frameworks. Throughout this process, every exported data point remains traceable to its original source entry, and dataset exports can be reliably reproduced given the same configuration and query database version. Because data is stored in a queryable, disk-backed format, ChemReporter can process datasets far larger than available memory, allowing it to scale to billion-structure datasets on standard compute infrastructure. ChemReporter is available on GitHub and PyPI under the Apache License 2.0.