Search papers, labs, and topics across Lattice.
This paper introduces ECLAIR, a causally grounded AI framework that integrates Large Language Models (LLMs) into the scientific process of empirical software engineering, enhancing hypothesis generation, analysis, and interpretation. By treating LLMs as active scientific agents and employing a human-in-the-loop design, the framework mitigates risks associated with unsound automated reasoning. A case study reveals that certain prompt designs negatively impact code generation accuracy, demonstrating how causal reasoning can elucidate the underlying mechanisms of software phenomena.
Causal reasoning reveals that specific prompt designs can detrimentally affect code generation accuracy in LLMs, challenging assumptions about optimal input strategies.
The scientific method has long guided empirical research in Software Engineering (SE), but the complexity of modern software systems often hinders its systematic application. This paper introduces _ECLAIR_, a causally grounded AI framework that integrates Large Language Models (_LLMs_) into every stage of the scientific process, from hypothesis generation to analysis and interpretation. _ECLAIR_ treats _LLMs_ as active **scientific agents** operating under the principles of causal inference, within a human-in-the-loop design that safeguards against the risks of unsound automated reasoning. We demonstrate the framework through a case study examining how prompt design influences code generation accuracy in two _LLMs_. Results show that, for both models, instruction-style, longer few-shot, and signature-augmented prompts yield small negative causal effects on accuracy, illustrating how causal reasoning provides a principled foundation for explaining _why_ software phenomena occur. This study presents the first causally grounded structured methodology for embedding _LLMs_ within the scientific method in SE, designed around the epistemological demands of empirical SE research, establishing a basis for rigorous AI-assisted research.
Microsoft Research