Search papers, labs, and topics across Lattice.
This paper introduces GeoDe, a novel abstention fine-tuning framework that addresses hallucination in LLMs by explicitly modeling and denoising the "gray zone" of ambiguous knowledge boundaries in the latent space. GeoDe uses linear probes to construct a truth hyperplane and employs geometric distance from this hyperplane as a confidence signal for filtering noisy samples during fine-tuning. Experiments on Llama3 and Qwen3 show that GeoDe significantly improves truthfulness and generalization across various benchmark datasets.
LLMs' hallucinations stem from a "gray zone" of internal belief ambiguity near knowledge boundaries, and geometric denoising in the latent space offers a surprisingly effective way to purge it.
Large language models (LLMs) often exhibit hallucinations due to their inability to accurately perceive their own knowledge boundaries. Existing abstention fine-tuning methods typically partition datasets directly based on response accuracy, causing models to suffer from severe label noise near the decision boundaries and consequently exhibit high rates of abstentions or hallucinations. This paper adopts a latent space representation perspective, revealing a"gray zone"near the decision hyperplane where internal belief ambiguity constitutes the core performance bottleneck. Based on this insight, we propose the **GeoDe** (**Geo**metric **De**noising) framework for abstention fine-tuning. This method constructs a truth hyperplane using linear probes and performs"geometric denoising"by employing geometric distance as a confidence signal for abstention decisions. This approach filters out ambiguous boundary samples while retaining high-fidelity signals for fine-tuning. Experiments across multiple models (Llama3, Qwen3) and benchmark datasets (TriviaQA, NQ, SciQ, SimpleQA) demonstrate that GeoDe significantly enhances model truthfulness and demonstrates strong generalization in out-of-distribution (OOD) scenarios. Code is available at https://github.com/Notbesidemoon/GeoDe.
