Search papers, labs, and topics across Lattice.
This study investigates the robustness of AI research agents in materials science by assessing whether selected modeling changes can generalize to unseen data and be reused across different tasks. Through a series of seven searches, the authors evaluate 701 changes across ten Matbench endpoints, revealing that nine out of ten interventions maintain their effectiveness on held-out data. The findings highlight the potential for closed-loop agents to yield reusable code and decisions that withstand external validation, with significant improvements in material property predictions such as a 26.3% mean reduction in error for combined feature and model changes.
Nine out of ten AI-selected modeling changes in materials science remain effective when tested on unseen data, showcasing the potential for reusable AI-driven discoveries.
An AI research agent can improve the score it sees without finding a modelling change that works on new materials. We ask a stricter question. After repeated experiments, does the selected change survive on data that never entered the loop, and can its code be reused? We separate the search into changes to features, models, representations, and training data. Seven searches produce 701 evaluated changes across ten Matbench endpoints. Agents receive only the mean over five inner folds, reducing reliance on any single development split. We then freeze the selected code and evaluate it once on an untouched holdout. Nine of ten choices remain the best tested single intervention. The surviving changes reveal two materials modelling regimes. With composition alone, feature, model, and representation changes provide comparable routes to improvement. They include held-out MAE reductions of 17.4\% for band gap and 18.6\% for steel strength, as well as gains on both classification endpoints, while screened external data adds little. For structure tasks, richer geometry descriptors and model or calibration changes lower mean held-out MAE by 14.6\% and 7.1\% and lead on different property families, whereas composition embeddings do not transfer. Combining separately found feature and model changes yields a 26.3\% mean held-out improvement. These results show in materials prediction that closed-loop agents can produce decisions that survive unseen evidence and code changes that can be reused across tasks and combined. More broadly, they provide an evaluation design for testing executable discoveries beyond the feedback loop.
CMU ML