Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The paper introduces CausalFlip, a new benchmark designed to evaluate the causal reasoning abilities of LLMs by creating semantically similar question pairs that require opposite causal judgments based on underlying causal structures (confounder, chain, collider). CausalFlip includes a noisy-prefix evaluation to further test models' reliance on semantic patterns by adding irrelevant text to intermediate reasoning steps. Experiments demonstrate that while Chain-of-Thought (CoT) prompting can be misled by semantic correlations, an internalized causal reasoning approach yields better causal grounding, suggesting potential for improving latent causal reasoning in LLMs.
LLMs can be tricked into making incorrect causal judgments by semantically similar questions, even with Chain-of-Thought prompting, highlighting a critical gap in true causal reasoning.
As large language models (LLMs) witness increasing deployment in complex, high-stakes decision-making scenarios, it becomes imperative to ground their reasoning in causality rather than spurious correlations. However, strong performance on traditional reasoning benchmarks does not guarantee true causal reasoning ability of LLMs, as high accuracy may still arise from memorizing semantic patterns instead of analyzing the underlying true causal structures. To bridge this critical gap, we propose a new causal reasoning benchmark, CausalFlip, designed to encourage the development of new LLM paradigm or training algorithms that ground LLM reasoning in causality rather than semantic correlation. CausalFlip consists of causal judgment questions built over event triples that could form different confounder, chain, and collider relations. Based on this, for each event triple, we construct pairs of semantically similar questions that reuse the same events but yield opposite causal answers, where models that rely heavily on semantic matching are systematically driven toward incorrect predictions. To further probe models' reliance on semantic patterns, we introduce a noisy-prefix evaluation that prepends causally irrelevant text before intermediate causal reasoning steps without altering the underlying causal relations or the logic of the reasoning process. We evaluate LLMs under multiple training paradigms, including answer-only training, explicit Chain-of-Thought (CoT) supervision, and a proposed internalized causal reasoning approach that aims to mitigate explicit reliance on correlation in the reasoning process. Our results show that explicit CoT can still be misled by spurious semantic correlations, where internalizing reasoning steps yields substantially improved causal grounding, suggesting that it is promising to better elicit the latent causal reasoning capabilities of base LLMs.
Johns Hopkins