Search papers, labs, and topics across Lattice.
The paper identifies a "Causal Tongue-Tie" phenomenon in LLMs, where internal representations encode the correct causal answer to anti-commonsense questions, but the final Yes/No output reverts to the commonsense answer. Using a linear probe on hidden states, the authors achieve high accuracy (~0.97) in recovering the evidence-supported answer, while direct Yes/No outputs show chance performance (~0.5). This discrepancy reveals a disconnect between the model's internal causal understanding and its verbal expression.
LLMs might understand causality better than their Yes/No answers suggest, meaning current benchmarks could be misleadingly pessimistic.
We find a mismatch between what large language models encode about a causal question and what they answer. On anti-commonsense CLadder items, a fixed linear probe recovers the evidence-supported answer from the model's hidden state (accuracy approximately 0.97), while the spoken Yes/No reverts to the commonsense one (accuracy approximately 0.5). We call this approximately +0.5 gap Causal Tongue-Tie: a wrong Yes/No decomposes into two separable failure modes: no internal signal versus a signal the verbal interface cannot say. The implication cuts both ways for output-only causal benchmarks: a benchmark "correct" need not mean the model has understood, and a benchmark "wrong" need not mean it cannot. Sweeping claims about whether LLMs can do causal reasoning, drawn from a single accuracy number, deserve a second look.
