Search papers, labs, and topics across Lattice.
This study evaluates the reasoning capabilities of human participants and 25 large language models (LLMs) in common-sense scenarios, revealing that both exhibit similar error patterns, suggesting a shared reliance on pattern matching rather than abstract reasoning. By analyzing attention heads in LLMs, the researchers demonstrate that these heads facilitate a form of pattern-matching that can predict reasoning errors in humans linked to irrelevant prompt details. The findings challenge the notion that human reasoning is fundamentally distinct from LLM behavior, highlighting a shared mechanism in everyday causal reasoning.
Human reasoning mirrors LLMs in its reliance on pattern matching, revealing surprising commonalities in error patterns across both domains.
When large language models (LLMs) fail to generalize or make haphazard errors in reasoning, it is often taken as evidence that LLMs are not truly reasoning, but rather performing a kind of pattern matching. The implication is that people's behavior does not exhibit the same types of failures because human reasoning uses principled and abstract world models. We evaluate human participants and 25 LLMs on their ability to engage in common-sense reasoning about a variety of everyday situations and observe similar patterns of errors in both people and models. We then identify the set of attention heads driving LLM responses and find that these heads implement a form of pattern-matching. These attention heads allow us to predict seemingly inexplicable reasoning errors in people caused by ostensibly irrelevant prompt details. Taken together, our results suggest that everyday causal reasoning in people and LLMs is more consistent with a form of pattern-matching than with abstract world models.
