Search papers, labs, and topics across Lattice.
This paper critiques the foundational assumptions of software measurement, highlighting the shift from manual data collection to leveraging traces from version control systems, particularly in the context of open-source software. It emphasizes that the increasing use of AI agents in software development has led to data that may not originate from human developers, thus challenging the validity of existing measurement methodologies. The authors propose a systematic AI-assisted replication program to reassess key findings and adapt measurement techniques to ensure their relevance in the evolving landscape of software development.
The rise of AI in software development is rendering traditional software measurement assumptions obsolete, necessitating a radical rethink of how we validate our findings.
For most of the past six decades, software measurement relied on labor-intensive manual collection of proprietary data, which hampered progress. The shift to repurposing traces from version control and related tools dramatically expanded data availability$\unicode{x2014}$especially with the rise of open-source software$\unicode{x2014}$but hinged on an often unstated assumption: that these tools are used by professional developers to build genuine software systems. However, as trace-generating tools, data types and scale, and empirical methods have all evolved, it has become clear that changes in data generation and analytical approaches affect many prior findings about software development, maintenance, and evolution. With AI agents now actively using these same tools, the resulting traces frequently violate the original assumption of human origin. To preserve the relevance of software measurement research, immediate action is needed: We must detect when foundational assumptions are violated in contemporary data and develop new methodologies that remain valid under changed circumstances. To this end, we propose a systematic AI-assisted replication program that revisits key findings using modern techniques, aiming for methods that yield consistent results on current data to keep software measurement meaningful.
CMU ML