Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study investigates the impact of knowledge alignment in supervised fine-tuning (SFT) of language models, specifically addressing how misalignment can lead to factual hallucinations. By introducing two novel methods—Evidence Rewrite and Recall Rewrite—alongside existing techniques, the authors demonstrate that constraining SFT targets to the model's internalized knowledge significantly reduces hallucinations while maintaining overall performance. The results indicate that SFT targets that exceed the model's knowledge contribute to hallucination rates, with Recall Rewrite showing the most substantial improvements in factual accuracy and refusal behavior.
Knowledge-aligned fine-tuning can drastically cut down on hallucinations in language models while preserving their general capabilities.
Supervised fine-tuning (SFT) trains a base language model to imitate target responses, and these targets may require knowledge the base model has not robustly internalized. We study this as a source of hallucinations and frame a group of mitigation methods as \emph{knowledge-aligned SFT}: constraining SFT training targets to the base model's parametric knowledge. Under a unified setup, we compare existing generation-based and estimation-based knowledge-alignment methods and introduce two new variants: Evidence Rewrite, which verifies base-model generations using external evidence, and Recall Rewrite, which retains claims only when they can be consistently recalled by the base model. Experiments with Qwen 3 4B and OLMo 3 7B show that knowledge-aligned SFT can reduce factual hallucinations on WildHalu and Biography while largely preserving general capabilities. Recall Rewrite yields the strongest factuality gains and improves refusal behavior on UnknownBench. It thereby confirms that SFT targets beyond the base model's knowledge drive hallucination behavior.