Search papers, labs, and topics across Lattice.
This study analyzes documentation practices in AI research by examining 56,800 conference papers from five leading venues over the past decade. The findings indicate a significant improvement in reproducibility, with the percentage of papers sharing both code and data rising from 11% to 64%, and inferred reproducibility rates increasing from 28% to 64%. Notably, these enhancements in documentation practices appear to stem from a broader shift towards open science rather than solely from the implementation of reproducibility checklists.
Reproducibility in AI research has surged from 28% to 64% in just a decade, driven by a cultural shift towards open science.
The reproducibility crisis has directed the AI research community toward improving documentation practices. Several studies have identified methodological issues, and in response, the most impactful venues in the field have introduced reproducibility checklists. We seek to understand whether documentation practices have changed over time by assessing all published papers at five leading AI conferences over the past decade. Seven reproducibility variables were identified, quality-assured and used to analyse 56 800 publications. Our analysis reveals that in the period 2014 to 2024, documentation practices have improved; papers sharing both code and data increased nearly sixfold, from 11% to 64% Building on empirical reproducibility rates from a prior study, we estimate - inferred from documentation practices, not direct testing - that reproducibility increased from 28% in 2014 to 64% in 2024. Improvements in documentation practices predate the introduction of reproducibility checklists, suggesting these changes reflect a broader movement toward open science rather than a direct response to formal requirements.
