Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study investigates the impact of different representational formats—natural language, logical notation, linear diagrams, and Euler diagrams—on the syllogistic reasoning capabilities of two large language models, Claude 3.5 Sonnet and GPT-4o-mini. Despite previous findings suggesting that diagrams enhance human reasoning, the results indicate that these models do not consistently benefit from diagrammatic representations, particularly struggling with neutral problems and exhibiting systematic conversion errors. The findings highlight the limitations of current LLMs in leveraging visual aids for logical reasoning, raising questions about their reasoning capabilities in complex tasks.
Diagrammatic representations fail to provide a consistent advantage for LLMs in syllogistic reasoning, challenging assumptions about their utility in AI reasoning tasks.
Diagrams are widely used to support logical reasoning, and prior studies suggest that representations such as Euler diagrams can improve human reasoning performance. Recent work has also explored their effects on large language models (LLMs). In this paper, we compare four representational conditions for syllogistic reasoning: natural language, logical notation, linear diagrams, and Euler diagrams. Using 285 problems from Ando et al. (2024), we evaluate two contemporary LLMs, Claude 3.5~Sonnet and GPT-4o-mini. Our results show that diagrammatic representations do not consistently improve performance. Although the models perform well on entailment and contradiction problems, they struggle with neutral problems and often make systematic conversion errors. Overall, the results suggest that the tested models gain limited benefit from diagrams in logical reasoning tasks.