Search papers, labs, and topics across Lattice.
This study investigates how large language models (LLMs) perform in active multi-turn information acquisition for abductive reasoning using the Alien Abduction game, which tests their ability to form and revise hypotheses based on evidence. The results reveal that LLMs show significantly higher success rates when evidence is provided upfront rather than distributed across turns, and they struggle with hypothesis validation and refinement in multi-turn interactions. Notably, models tend to commit to hypotheses based on self-selected evidence, leading to inconsistencies and a lack of convergence in their reasoning processes.
LLMs often commit to flawed hypotheses based on self-selected evidence, highlighting critical gaps in their abductive reasoning capabilities.
Abductive reasoning requires forming hypotheses that explain observed evidence and revising them as new evidence becomes available. While large language models (LLMs) are often evaluated on whether they solve abductive reasoning tasks correctly, less is known about how they acquire evidence, update their hypotheses, and decide when to stop. We introduce Alien Abduction game, an interactive probe for studying these behaviours under different interaction modes. The modes vary in whether evidence is provided upfront or across turns, and whether queries are selected by the model or examples are provided by the oracle. Across models, providing evidence upfront leads to higher success rates than distributing it across turns. In multi-turn settings, some models commit before using the available evidence, while others exhaust the turn budget without converging. Models also achieve higher success rates when examples are provided by the oracle than when they select their own queries, although their final hypotheses are more consistent with the evidence they selected. These findings suggest that models may form hypotheses that fit self-selected evidence without sufficiently distinguishing them from alternatives, and may struggle to validate and refine their hypotheses or determine when to stop.
