Search papers, labs, and topics across Lattice.
This meta-study investigates the state of replication practices in the emerging field of usable security and privacy by analyzing 24 user study replications published from 2016 to 2025. Utilizing a mixed-method approach, the authors identify key challenges such as the lack of clear guidelines in Calls for Papers and the prevalence of modifications in replication studies, which complicate the assessment of fidelity to original research. The findings underscore the need for improved standards and recommendations to enhance replication practices in this domain, ultimately fostering more reliable research outcomes.
Replication studies in usable security and privacy often modify original research significantly, raising questions about their validity and the standards of the field.
The field of usable security and privacy research is a young and expanding field, which is still developing standards for its research, e.g. regarding replications. We used a mixed-method approach, in order to get a better understanding of the current state of replications in the field of usable security and privacy: (1) we examine the Call for Papers of 13 venues spanning security, privacy, and human-computer interaction; (2) we conduct a systematic search for papers reporting replicated user studies published across these venues between 2016 and 2025, yielding 24 relevant publications; (3) we categorized these 24 papers employing the replication taxonomy proposed by Olszewski et al. (2025); (4) we distributed a survey to the authors of these papers to understand their motivations for conducting replications. Our analysis reveals four key insights: (A) Calls for Papers would benefit from clearer guidelines for authors and reviewers regarding replication work; (B) determining what modifications were made relative to the original study proves difficult when reading replication papers; (C) strict exact replications do not exist in our sample. Approximately two-thirds of the 24 studies altered multiple aspects of the original work; (D) temporal and contextual changes affecting results emerged as one of the most frequently cited motivations for replication. Based on these findings, we offer practical recommendations for venues, researchers, and peer reviewers to strengthen replication practices in usable security and privacy research.
