Search papers, labs, and topics across Lattice.
This study investigates the flexible reasoning capabilities of large language models (LLMs) compared to humans using a novel "riddle riddle" paradigm, which tests the ability to apply appropriate reasoning strategies to word problems. The results reveal that LLMs significantly outperform humans on traditional riddles (84.9% vs. 50.5%) but struggle with riddle riddles, achieving only 50.7% accuracy, indicating a reliance on surface features rather than flexible reasoning. In contrast, humans excel at riddle riddles (80.5%) but perform poorly on genuine riddles, suggesting that LLMs may primarily rely on memory retrieval rather than true reasoning flexibility, raising concerns about misinterpreting LLM outputs as genuine reasoning.
LLMs excel at traditional riddles but falter dramatically when faced with riddle riddles, revealing a critical limitation in their reasoning flexibility.
Humans flexibly adapt their reasoning strategies to the requirements of a given problem. Large language models (LLMs) have performed well on many cognitive tasks, however, it is unclear whether this accuracy is a result of pattern matching from training data or flexible reasoning. Here, we introduce a novel paradigm to test this question: the riddle riddle paradigm. Riddle riddles are word problems written to mimic popular riddles, but altered so their answers only require literal interpretations. Identifying correct answers requires looking past the structure of each question and flexibly apply different reasoning strategies based on the content. If LLMs respond to surface features, such as form, a riddle-like structure should cause models to use an inventive reasoning strategy even when a literal interpretation suffices. Alternatively, if LLMs reason based on content, they should flexibly switch strategies when appropriate. Across two experiments with nine state-of-the-art LLMs and 100 human participants, we show humans and LLMs fail on this paradigm in opposite directions. LLMs were far more accurate on genuine riddles than on riddle riddles (84.9% vs. 50.7%); whereas humans showed the reverse effect (50.5% vs. 80.5%). Error analysis shows that 90.8% of LLM errors on riddle riddles (the condition where they show diminished performance) were due to inappropriate use of inventive reasoning while only 57.6% of human errors on genuine riddles were due to overextending literal reasoning. Thus, while both groups make mistakes, reasoning mistakes are made more often by LLMs than by humans. Overall, LLMs' strong performance on genuine riddles may reflect memory retrieval rather than flexible strategy selection, and without stimuli designed to elicit this contrast, it becomes easy to conflate LLM-generated outputs that look like reasoning with genuine reasoning.
