Search papers, labs, and topics across Lattice.
This paper introduces the Artificial Intelligence (AI)-based Epidemiology Research Assistant (ARA), a framework designed to enhance the reliability of automated research systems by explicitly encoding causal design principles and methodological constraints. ARA translates natural language research questions into structured causal protocols and generates synthetic datasets using Structural Causal Models (SCMs), which allows for controlled evaluation of causal assumptions. While the integration of protocol construction and adversarial validation did not consistently improve numerical accuracy, it significantly shifted the failure mode from silent errors to the identification of protocol concerns and diagnostic failures, highlighting the importance of transparency in causal inference.
ARA reveals that automated research systems can expose hidden protocol issues rather than just produce misleading causal estimates.
While automated research systems promise to accelerate empirical analysis, they are prone to silent failures: instances in which analysis code executes successfully yet relies on invalid causal assumptions. We present the Artificial Intelligence (AI)-based Epidemiology Research Assistant (ARA), a framework that makes these failures visible by explicitly encoding causal design principles, study-specific assumptions, and methodological constraints. ARA integrates protocol construction, synthetic data generation, and adversarial validation into a unified pipeline. The framework translates natural language research questions into structured causal protocols and executable analysis code by first constructing a protocol and then generating synthetic datasets using Structural Causal Models (SCMs) with known ground-truth effects. This synthetic-data step can also support pipeline development when access to confidential data, such as medical data, is restricted. The generated analysis is then evaluated under controlled violations of identification assumptions. We evaluate ARA on the Automated Causal Reasoning Benchmark, assessing recovery of identification strategies, causal quantities, treatment and outcome variables, and consistency between generated code and approved protocol. Protocol construction and adversarial validation did not consistently improve numerical agreement with benchmark estimates compared with standard LLM-based generation. However, they changed the failure mode: instead of silently returning causal estimates, ARA often surfaced protocol concerns, diagnostic failures, incomplete inference, or downgraded non-causal interpretations. These findings suggest that validity-first automated science systems should be evaluated not only by answer accuracy, but also by whether they indicate when causal claims are unwarranted.
