Search papers, labs, and topics across Lattice.
This study introduces the Controlled Reformulation Testing benchmark (CRTBench), comprising 350 question families to assess the logical consistency of large language models (LLMs) across various reformulations. The evaluation reveals a significant accuracy-consistency gap, with GPT-5.4-mini achieving 98.9% accuracy but only 60.3% consistency across reformulations, particularly struggling with logically complex transformations like contrapositive rewriting. The findings underscore that high accuracy does not equate to reliable logical reasoning, highlighting the need for more nuanced evaluation metrics in LLMs.
LLMs can achieve high accuracy while failing to maintain logical consistency, revealing a critical gap in their reasoning capabilities.
Large language models (LLMs) frequently contradict themselves when the surface form of a logically equivalent question changes. We present a benchmark of 350 question families (1,750 total questions) for Controlled Reformulation Testing (CRTBench) to evaluate logical invariance. In this benchmark, we investigate LLMs'ability to maintain consistent answers across controlled reformulations, which include contrapositive rewriting, double negation, negation flipping, and passive voice. We evaluate several frontier LLMs and observe an accuracy-consistency gap where GPT-5.4-mini achieves $98.9\%$ base accuracy but only $60.3\%$ family-level consistency, while reasoning-optimized o4-mini achieves $96.9\%$ consistency. From our experiments, we observe that failures cluster around logically nontrivial transformations such as contrapositive rewriting ($72.4\%$ for GPT-5.4-mini) and double negation ($84.6\%$), while surface-level rephrasing remains robust ($94-100\%$). Increasing reasoning effort improves GPT-5.4-mini to $85.4\%$ consistency, but leaves GPT-5.4 unchanged overall because gains on nested negation are offset by failures on quantifier families. These results show that accuracy alone is not enough for evaluating logical reasoning in LLMs.
