Search papers, labs, and topics across Lattice.
CausaLab is introduced as a scalable environment for evaluating LLM agents in interactive causal discovery, requiring agents to both predict outcomes and recover the underlying causal mechanisms. Experiments using GPT-5.2-high show a significant gap between prediction accuracy (92%) and causal structure recovery (F1 score of 0.471) in observational settings, highlighting the difficulty of inferring causal relationships. Mixed observation-intervention strategies improve structural fidelity, but agents struggle to design truly informative interventions, and premature stopping further hinders performance.
LLMs can predict the outcome of causal experiments with high accuracy, but often fail to correctly identify the underlying causal mechanisms, revealing a critical gap in their ability to reason about cause and effect.
We introduce CausaLab, a scalable environment for evaluating interactive causal discovery by LLM agents. Unlike prior evaluations, CausaLab evaluates both whether an agent can solve a problem using causal evidence and whether its answer is supported by a correct hypothesis about the underlying causal mechanism. Each episode places an agent in a synthetic laboratory: it receives prior measurement records, intervenes on a manipulator crystal, and predicts the resonance frequency of a held-out reactor crystal governed by the same mechanism. The hidden data-generating process is a randomly sampled structural causal model (SCM), so success requires recovering both a causal graph and structural equations rather than recalling prior knowledge. CausaLab also includes a domain-specific language that records the agent's evolving SCM hypothesis, making trajectories inspectable and comparable with ground truth. Experiments show a persistent gap between prediction and mechanism recovery: in the purely observational 6-node setting, GPT-5.2-high reaches 92% task accuracy but only 0.471 all-edge $F_1$. This observation further motivates our exploration of different interaction strategies: Mixed observation--intervention strategies improve structural fidelity: in the mixed 6-node setting, GPT-5.2-high achieves 80% on both task accuracy and all-edge $F_1$. Yet even strong agents struggle to design informative interventions, as pure intervention strategies perform poorly on both task accuracy and all-edge $F_1$. We identify premature stopping as a major weakness of agents, and show that asking the model to verify the consistency between its hypothesis and past data can help mitigate this issue. CausaLab therefore separates predictive success from causal understanding and exposes current LLM agents' limits as experimental causal reasoners.
CMU ML