Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
InterveneBench is introduced as a novel benchmark to evaluate LLMs' ability to perform intervention-centered research design reasoning in real-world social science contexts, using 744 peer-reviewed studies across diverse policy domains. The benchmark requires LLMs to reason about policy interventions and identification assumptions without predefined causal graphs or structural equations. Results show that current LLMs struggle with InterveneBench, but a proposed multi-agent framework called STRIDES significantly improves performance.
LLMs still fall short when it comes to reasoning about real-world policy interventions and causal study design, as revealed by the new InterveneBench benchmark.
Causal inference in social science relies on end-to-end, intervention-centered research-design reasoning grounded in real-world policy interventions, but current benchmarks fail to evaluate this capability of large language models (LLMs). We present InterveneBench, a benchmark designed to assess such reasoning in realistic social settings. Each instance in InterveneBench is derived from an empirical social science study and requires models to reason about policy interventions and identification assumptions without access to predefined causal graphs or structural equations. InterveneBench comprises 744 peer-reviewed studies across diverse policy domains. Experimental results show that state-of-the-art LLMs struggle under this setting. To address this limitation, we further propose a multi-agent framework, STRIDES. It achieves significant performance improvements over state-of-the-art reasoning models. Our code and data are available at https://github.com/Sii-yuning/STRIDES.