Search papers, labs, and topics across Lattice.
This paper introduces CausalSE, a framework leveraging Structural Causal Models (SCMs) to improve causal inference in Empirical Software Engineering (ESE), addressing limitations of traditional statistical methods. They apply CausalSE to the Galeras dataset, analyzing the causal effect of prompt engineering strategies on code generation outcomes in GPT-3. Results show that associational analyses often overestimate the impact of interventions due to confounding bias, which SCMs can mitigate, leading to more accurate conclusions about treatment effects.
Naive statistical analyses can lead to false positives in software engineering experiments, as demonstrated by the overestimation of prompt engineering's impact on code generation when confounding bias isn't addressed.
Causal Inference offers a fundamental approach for advancing empirical software engineering (ESE) beyond traditional statistical association, enabling researchers to rigorously identify and quantify causal relationships in software experiments. This paper introduces CausalSE, a framework that operationalizes Judea Pearl's causal inference paradigm in ESE context. The paper focuses on Structural Causal Models (SCMs) to address the limitations of classical statistical methods in mitigating confounding bias. Through a case study using the Galeras dataset and propensity score matching, we demonstrate how CausalSE disentangles the effect of prompt engineering strategies on code generation outcomes in a popular LLM (i.e., GPT-3). The results reveal that while associational analyses can suggest improvements in certain interventions (e.g., more complex prompts), causal analysis often does not find a significant treatment effect, highlighting the risk of false positives when confounding is not addressed. By providing a tutorial-based methodology and a real-world case study, this work equips software researchers with practical tools to design, analyze, and interpret software experiments with methodological rigor, ultimately enabling more informed and actionable conclusions in both research and practice.
