Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
CausaLab, a new environment, evaluates LLM agents on interactive causal discovery by requiring them to both predict outcomes and recover the underlying causal mechanisms in a synthetic laboratory setting. Experiments reveal a significant gap between prediction accuracy (92% for GPT-5.2-high in a 6-node setting) and causal structure recovery (0.471 all-edge F1), indicating that LLMs can predict without truly understanding causal relationships. The study also finds that mixed observation-intervention strategies improve structural fidelity, and that consistency verification can mitigate premature stopping.
LLMs can ace prediction tasks in causal environments, but still fail to grasp the underlying causal mechanisms, revealing a critical blind spot in their reasoning abilities.
We introduce CausaLab, a scalable environment for evaluating interactive causal discovery by LLM agents. Unlike prior evaluations, CausaLab evaluates both whether an agent can solve a problem using causal evidence and whether its answer is grounded in a faithful recovered causal mechanism. Each episode places an agent in a synthetic laboratory: it receives prior measurement records, intervenes on a manipulator crystal, and predicts the resonance frequency of a held-out reactor crystal governed by the same mechanism. The hidden data-generating process is a randomly sampled structural causal model (SCM), so success requires recovering both a causal graph and structural equations rather than recalling prior knowledge. Experiments show a persistent gap between prediction and mechanism recovery: in the purely observational 6-node setting, GPT-5.2-high reaches 92% task accuracy but only 0.471 all-edge $F_1$. Mixed observation-intervention strategies improve structural fidelity, while pure intervention remains difficult even for strong agents. We identify premature stopping as a major weakness and show that consistency verification mitigates it. CausaLab therefore separates predictive success from causal understanding and exposes current LLM agents'limits as experimental causal reasoners.
CMU ML