Search papers, labs, and topics across Lattice.
This study investigates the effectiveness of existing validation workflows in detecting reproducibility-relevant mutations in machine learning research repositories using a tool called MLReproMutate. By applying controlled mutations related to random seeds, dependency versions, data partitioning, and evaluation configurations to 39 frozen repository cases, the authors found that only 8.7% of these mutations were detected by the current validation methods. The results highlight a significant gap in the ability of existing workflows to safeguard against changes that could impact reproducibility, suggesting a need for enhanced testing approaches in machine learning research.
Only 8.7% of reproducibility-relevant mutations in ML repositories are detected by current validation workflows, revealing a critical oversight in safeguarding research integrity.
Reproducibility in machine-learning research depends on experimental choices such as random seeds, dependency versions, data partitioning, and evaluation configuration. Existing repository validation workflows may execute successfully without detecting changes to such choices. We study this problem using MLReproMutate, research software that applies controlled, reproducibility-relevant mutations to ML research repositories and evaluates them against validation workflows already present in those repositories. We conducted an outcome-blind empirical study of 39 frozen repository-operator cases using four mutation classes: random seed, dependency pin, data split, and cross-validation fold count. Repository revisions, mutation candidates, and validation workflows were fixed before mutation outcomes were observed. Primary execution yielded outcomes for 13 of 39 cases; a bounded restoration procedure increased the combined evaluable set to 24. After excluding one confirmed-equivalent mutation, 23 confirmed non-equivalent mutations remained. The selected validation workflows detected 2 of these 23 mutations, corresponding to an observed detection proportion of 8.7%. These results do not imply that the corresponding repositories are irreproducible. Rather, they show that, in this sample, existing validation workflows often did not detect the particular controlled reproducibility-relevant changes introduced by the study. The findings motivate reproducibility-oriented mutation testing as a complementary way to assess whether research-software safeguards constrain experimentally important choices.
