Track trials, findings, and clinical evidence as it lands.
Search papers, labs, and topics across Lattice.
CausalVerify establishes an execution-grounded benchmark evaluating LLMs on end-to-end econometric causal estimation workflows across 259 published economics papers and 100 executable synthetic datasets spanning DiD, IV, RDD, and event studies. Unlike traditional benchmarks that stop at syntactic execution or text-level method choice, this work isolates whether model-generated R scripts accurately recover target treatment effects. Across seven evaluated LLMs, 15.5% of code that executed without error returned statistically incorrect causal estimates, with model pass rates ranging from 10% to 88% and self-reported confidence failing to track estimation accuracy.
Over 15% of LLM-generated causal inference pipelines run without a single code error yet return completely bogus treatment effects, proving that code execution is a dangerously misleading proxy for statistical validity.
Existing causal-inference benchmarks for LLMs mostly score method descriptions or whether generated code runs, not whether the executed workflow recovers the target causal estimate. CausalVerify studies this verification problem for structured econometric causal-estimation workflows by separating realistic interpretation from verifiable computation. It pairs 259 published economics papers (reconstructed research question, data description, institutional context) with 100 fixed-seed synthetic scenarios that realise CSV datasets for difference-in-differences, event study, instrumental variables, and regression discontinuity designs. Experiment A (real-paper text agreement) scores method-family and direction agreement against four-LLM consensus labels. Experiment B (synthetic execution) runs model-written R code and checks whether the extracted treatment-effect estimate matches a canonical estimator on the same realised dataset; this execution-grounded correctness layer is L2b+, distinct from L2b, which records only whether the code executes. A calibration arm asks whether self-reported confidence separates correct from incorrect workflows. On Experiment B, seven LLMs reach L2b+ pass rates of 10% to 88% at the default 50% tolerance, and 66 of the 426 workflows that execute (15.5%) return a wrong estimate. Execution ranking (L2b) agrees with L2b+ far better than text-direction scoring (L4): Kendall $τ=0.81$ and Spearman $ρ=0.93$, versus Kendall $τ$ between $-0.20$ and $0.10$ for L4. Llama-3.3-70B-Instruct shows the same qualitative gap, and reported confidence does not reliably separate correct from incorrect workflows. The claims are confined to standardized single-shot workflows in these four design families under the evaluated R backend and model panel; the benchmark does not measure general causal-inference ability. Code, data, cached outputs, and a datasheet are released.