Search papers, labs, and topics across Lattice.
This paper introduces the IAR (Inject, Align, Recover) framework, a three-stage post-training method designed to enhance retrieval-free question answering by internalizing document knowledge directly into large language models. The framework consists of injecting structured document knowledge, aligning the model with QA behavior, and recovering general capabilities, which collectively lead to significant improvements in domain-specific and general performance metrics. Experimental results demonstrate that IAR outperforms conventional methods, achieving an average increase of 3.6 percentage points in domain QA accuracy and 12.1 percentage points in general performance across multiple model families and datasets.
IAR achieves a remarkable boost in retrieval-free question answering, outperforming traditional methods by effectively internalizing document knowledge into LLMs.
Large language models often fail to answer questions about a bounded document collection when the source documents are not retrieved at inference time. We study this setting as document knowledge internalization: converting a fixed corpus into usable parametric knowledge for retrieval-free question answering. We propose IAR (Inject, Align, and Recover), a three-stage post-training framework that separates structured document knowledge injection, QA behavior alignment, and general ability recovery. Unlike conventional continued pretraining, Inject converts source documents into continuation, rewrite, and instruction-conditioned reconstruction objectives. Align then adapts the injected model with answer-only QA supervision, while Recover merges the domain-adapted model with the base instruction model to recover general capabilities. Across Common Corpus (CC) and CCI, and across Llama, Phi, Qwen, and SmolLM model families, IAR improves the domain-primary domain-general frontier for retrieval-free document internalization. In the main comparison, IAR improves over Vanilla SFT on all four reported metrics in 7 of 8 dataset-model settings, with average gains of 3.6 percentage points in domain QA accuracy and 12.1 percentage points in mean general performance across IFEval, MMLU, and MSBench. Extended CC baselines show that LoRA and FAPM can win individual general metrics, but among methods that also reach leading or near-leading domain internalization, IAR retains one of the strongest general profiles.
