Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study systematically reviews the assessment criteria used in recent replication studies within software engineering, revealing significant inconsistencies and a lack of shared standards. By analyzing ten selected studies, the authors identify issues such as contradictory criteria and insufficient acknowledgment of measurement uncertainty. They propose a principled framework based on statistical and methodological principles to enhance the evaluation of replication outcomes, ultimately aiming to improve the reliability of evidence accumulation in the field.
Inconsistent assessment criteria in software engineering replication studies lead to unreliable outcomes, but a new principled framework could change that.
Background: Replication studies in software engineering are increasingly common, yet their interpretation remains uncertain and inconsistent because assessments frequently rely on loosely defined or ad hoc criteria. Aim: This study aims to document how replication study outcomes are currently assessed in empirical software engineering, identify problems arising from inconsistent criteria and propose a principled framework for meaningful evaluation. Method: We conducted a systematic review of replication studies, with the search covering recent empirical software engineering replications (2021--2025). For each study, we extracted the criteria used to assess replication outcomes and analysed these for heterogeneity, logical consistency, and alignment with established statistical principles. Results: A total of 10 replication studies were located. The analysis reveals substantial heterogeneity in assessment practices, with contradictory criteria applied to similar data, limited acknowledgement of measurement uncertainty, and an absence of shared standards. We propose a principled framework grounded in statistical, methodological, and measurement considerations, and demonstrate its application through worked examples. Conclusions: Adopting consistent and transparent assessment principles would reduce ambiguity, improve comparability, and support more reliable evidence accumulation in software engineering replication research.