Search papers, labs, and topics across Lattice.
This study investigates the phenomenon of hallucinations in large language models, distinguishing between failures due to missing knowledge and those stemming from inference misalignment. By introducing a latent key-task model, the authors reveal how pretraining-frequency imbalances can lead to dominant shortcut paths that compromise constraint-sensitive reasoning. Their findings, validated through the TrapQA testbed, indicate that hallucinations often result from biased latent inference rather than a lack of knowledge, highlighting two specific failure modes in entity disambiguation and action selection.
Hallucinations in language models often stem from biased inference paths rather than simply missing information, revealing a deeper layer of reasoning failure.
Large language models often produce hallucinated answers that violate prompt-level constraints. A key diagnostic question is whether these failures reflect missing knowledge, or whether the model has the relevant information but follows the wrong inference path. We study this phenomenon as inference misalignment: a mismatch between the answer supported by the prompt and the answer favored by statistically salient latent associations. We formalize this view with a latent key-task model, in which pretraining-frequency imbalance can cause a shortcut path to dominate the constraint-sensitive path and induce positive inference loss. The framework predicts two failure modes: task-retrieval bias in entity disambiguation and key-selection bias in action choice. We introduce TrapQA, a controlled diagnostic testbed with two components. ScientistQA tests disambiguation among similar scientists with supplementary factual probes, while Real-Life Constrained QA tests everyday constraint following under salient shortcuts. Our results show that hallucination can arise from biased latent inference rather than absent knowledge alone.
