Search papers, labs, and topics across Lattice.
This paper introduces Knowledge-Weighted Fine-Tuning (KWFT), a method to mitigate hallucinations in LLMs by scaling the fine-tuning learning signal based on an instance-level knowledge score derived from multi-sampled inference. KWFT encourages models to abstain from answering out-of-scope queries, thereby reducing knowledge misalignment between pre-training and fine-tuning. Experiments demonstrate that KWFT enables models to more accurately express uncertainty and improve overall performance by discriminating between known and unknown instances.
LLMs can learn to say "I don't know" more effectively by weighting the fine-tuning signal based on how well they already know the answer.
While large language models (LLMs) demonstrate strong capabilities across diverse user queries, they still suffer from hallucinations, often arising from knowledge misalignment between pre-training and fine-tuning. To address this misalignment, we reliably estimate a fine-grained, instance-level knowledge score via multi-sampled inference. Using the knowledge score, we scale the learning signal according to the model's existing knowledge, while encouraging explicit "I don't know" responses for out-of-scope queries. Experimental results show that this approach allows the model to explicitly express uncertainty when it lacks knowledge, while maintaining accuracy on questions it can answer. Furthermore, we propose evaluation metrics for uncertainty, showing that accurate discrimination between known and unknown instances consistently improves performance.
