Search papers, labs, and topics across Lattice.
This study introduces AutoMatBench, an automatic optimization toolkit designed to enhance benchmarking for material property prediction (MPP) by addressing the limitations of existing tools like MatBench, particularly in evaluating out-of-distribution (OOD) performance. By integrating Bayesian optimization with the MatBench framework, AutoMatBench allows for a vast array of benchmarking configurations, revealing significant performance discrepancies across different setups and emphasizing the need for understanding causal effects. Results demonstrate that AutoMatBench can achieve comparable outcomes to traditional methods while reducing computational costs by over 50%, thereby accelerating the discovery of novel materials.
Performance discrepancies in material property prediction can be drastically reduced with AutoMatBench, which saves over half the computational cost while achieving similar results to established benchmarks.
Material property prediction (MPP) infers key properties from chemical composition and structure, accelerating the discovery and optimization of novel materials. In the realm of MPP, MatBench is a widely accepted benchmarking tool that defines over ten significant problems and provides the paradigm of performance evaluation for AI prediction models. Even though MatBench works well in benchmarking the performances of prediction models on in-distribution (ID) tasks and datasets, it lacks the ability to reflect their performances on out-of-distribution (OOD) material data, resulting failure in new material discovery. By combining the pipelines of MatBench and the existing researches on OOD performance evaluation, this study enables a huge space of benchmarking configurations, comprehensively reflecting the performances, abilities, and disadvantages of various AI prediction models. This work reports that the discrepancy of performances at different configuration values is huge and can be illustrated with prior knowledge and novel insights, therefore consideration of causal effect of configurations on performance results is necessary. In case of the impossibility of enumerative benchmarking at every configuration, this work further proposes AutoMatBench, an automatic toolkit with Bayesian optimization. Experiments with AutoMatBench reports that, within twelve steps of optimization, the similar results with MatBench and former OOD research can be accessed while more than half of the cost are saved. Besides, this tool also yields more essential findings on MPP benchmarking, positively contributing to the cost and efficiency of new material discovery.
