Search papers, labs, and topics across Lattice.
This study investigates the limitations of Large Language Models (LLMs) in implicit multi-hop reasoning, revealing that even when individual facts are memorized, models struggle to integrate them in a single query. By contrasting groups exposed to compositional contexts during pretraining with those that were not, the authors establish that the observed reasoning failures stem from pretraining deficiencies rather than a lack of knowledge. Their findings indicate that exposure to compositional contexts is essential for enabling LLMs to perform implicit multi-hop reasoning effectively, with data-centric augmentation methods showing promise for improving performance in exposed individuals.
Implicit multi-hop reasoning in LLMs is fundamentally constrained by pretraining exposure, not knowledge gaps.
Large Language Models fail at implicit multi-hop reasoning: a model answers "When was $X$ born?" and "Who is $Y$'s closest friend?" correctly but fails on "When was $Y$'s closest friend born?" in a single forward pass, even when both facts are perfectly memorized and individually retrievable. We study this failure in a controlled natural language setting with a strict separation between individuals exposed to compositional contexts during pretraining and those that never appear in any such context. We confirm that compositional failure persists even at 97% 1-hop accuracy, establishing the gap as a pretraining failure rather than a knowledge absence. We propose and test nine data-centric augmentation formats and find that compositional pretraining transfers to unseen questions for exposed individuals, but never to individuals absent from compositional pretraining, suggesting that exposure to compositional contexts during pretraining is a necessary condition for implicit multi-hop reasoning.
