Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study introduces Knowledge-less Language Models (KLLMs), which are pretrained on anonymized corpora to suppress parametric recall and enhance evidence-grounded reasoning. By reducing reliance on outdated or misaligned factual knowledge, KLLMs demonstrate improved performance on contextual question answering, fact verification, and hallucination detection, achieving up to 25% relative gains over traditional models in retrieval-grounded scenarios. The findings indicate that modifying the pretraining signal can lead to more reliable and robust language models that better utilize external evidence, thus shifting the paradigm of knowledge acquisition in AI.
Suppressing parametric recall in language models can lead to significant improvements in contextual reasoning and reliability, challenging the conventional reliance on factual knowledge.
Language models encode substantial factual knowledge in their parameters, which can lead to unreliable behavior when this knowledge is outdated, incomplete, or misaligned with the provided context. In this work, we study whether modifying the pretraining signal can systematically shift models away from parametric recall and toward evidence-grounded reasoning. We introduce Knowledge--''Less''Language Models (KLLMs), a fundamentally different epistemic training paradigm for LLMs, which are pretrained on corpora in which named entities are anonymized, thereby removing a primary channel for entity-linked factual supervision. This intervention substantially reduces closed-book factual recall, while often improving performance on tasks where relevant information is provided as context. Across multiple model scales, KLLMs consistently outperform matched baselines on contextual question answering, fact verification, and hallucination detection benchmarks. Crucially, in retrieval-grounded settings with imperfect evidence, KLLMs show improved robustness and achieve up to 20--25\% relative gains over standard language models. They further exhibit better calibration, with improved ECE, Brier score, and AUROC, as well as more reliable abstention behavior. Our results demonstrate that suppressing entity-linked supervision during pretraining induces a shift in epistemic behavior: KLLMs rely less on parametric knowledge and more on external evidence, leading to improved reliability under realistic conditions. This suggests that pretraining-time control over knowledge acquisition can complement retrieval-augmented and tool-based systems by providing a more evidence-sensitive base model.