Search papers, labs, and topics across Lattice.
This paper critiques the self-consistency method in causal reasoning for LLMs, highlighting its failure to reliably identify valid answers due to confounding errors and fragmented voting among multiple valid responses. To address this, the authors introduce CALVER, a training-free symbolic verifier that evaluates reasoning traces against established causal criteria, achieving a significant performance improvement over traditional methods. On CLEAR find-one-valid queries, CALVER outperforms existing approaches, reaching 42.1% accuracy compared to around 30% for other methods, demonstrating its robustness across various models and scenarios.
CALVER reveals that traditional voting mechanisms in LLMs can be misleading, achieving over 42% accuracy in identifying valid causal answers where others fail.
Self-consistency assumes the most frequent answer among sampled reasoning traces is the most reliable, but this can fail in causal reasoning: samples often repeat the same confounding error, and votes fragment across multiple valid answers, letting an invalid answer win despite a valid minority trace. We introduce CALVER (Causal Axiom-Level VERification), a training-free symbolic verifier that scores structured traces against Pearl's causal criteria, including -separation, backdoor adjustment, and intervention, and selects the highest-scoring candidate without consulting a reference answer. On CLEAR find-one-valid queries that admit multiple graph-valid answers, CALVER reaches 42.1% where plurality, a reward model, an LLM judge, and model confidence remain near 30% on identical frozen pools. Scaling the judge to 72B does not close the gap. In an audited clean-core subset, 11 of 21 graph-valid CALVER selections differ from the benchmark's listed answer while still satisfying the requested predicate. The advantage widens with the sampling budget and reproduces across ten published Bayesian networks, a second model family, and settings where the model must build the graph from text. CALVER also improves thresholded average-treatment-effect decisions against exact ground truth, generalizes to logic under a truth-table checker, and scores each candidate in milliseconds on CPU. CALVER needs only a causal structure, supplied outright or built from the text; wherever that holds, selection can aggregate via causal validity.
