Search papers, labs, and topics across Lattice.
This study systematically reviews the assessment practices of replication studies in software engineering, revealing significant inconsistencies and a lack of standardized criteria. The analysis of ten recent studies highlights issues such as contradictory evaluation methods and insufficient consideration of measurement uncertainty. To address these challenges, the authors propose a principled framework that aligns with established statistical principles, aiming to enhance the reliability and comparability of replication outcomes in the field.
Inconsistent assessment criteria in software engineering replication studies lead to unreliable outcomes, but a new principled framework could transform evaluation practices.
Background: Replication studies in software engineering are increasingly common, yet their interpretation remains uncertain and inconsistent because assessments frequently rely on loosely defined or ad hoc criteria. Aim: This study aims to document how replication study outcomes are currently assessed in empirical software engineering, identify problems arising from inconsistent criteria and propose a principled framework for meaningful evaluation. Method: We conducted a systematic review of replication studies, with the search covering recent empirical software engineering replications (2021--2025). For each study, we extracted the criteria used to assess replication outcomes and analysed these for heterogeneity, logical consistency, and alignment with established statistical principles. Results: A total of 10 replication studies were located. The analysis reveals substantial heterogeneity in assessment practices, with contradictory criteria applied to similar data, limited acknowledgement of measurement uncertainty, and an absence of shared standards. We propose a principled framework grounded in statistical, methodological, and measurement considerations, and demonstrate its application through worked examples. Conclusions: Adopting consistent and transparent assessment principles would reduce ambiguity, improve comparability, and support more reliable evidence accumulation in software engineering replication research.
