Search papers, labs, and topics across Lattice.
This paper investigates the use of Mixture-of-Experts (MoE) architectures to enhance hallucination detection in Large Language Models (LLMs) by enabling per-token localization of false content. The authors introduce InnerExpert, a novel method that utilizes unique internal signals from MoE architectures, such as router entropy and expert disagreement, to create compact feature vectors for effective classification. Results demonstrate that InnerExpert significantly outperforms existing detection methods, achieving high AUROC scores while maintaining efficiency with a single forward pass.
Harnessing the unique signals from Mixture-of-Experts architectures, InnerExpert achieves unprecedented accuracy in detecting hallucinations at the token level.
Despite their widespread use, Large Language Models (LLMs) remain limited by a fundamental problem: the generation of plausible but false content, known as hallucinations. Most existing detection methods operate at the answer or sentence level, yet per-token detection is essential for localizing hallucinated spans and enabling fine-grained interventions. In this paper, we explore the use of the Mixture-of-Experts (MoE) paradigm to address this gap. In MoE architectures, a single forward pass activates a sparse subset of experts (i.e., distinct feedforward networks per layer) via a routing mechanism, producing internal signals (e.g., router entropy, expert disagreement, and expert usage patterns) that are unavailable in dense architectures and have not been previously exploited for hallucination detection. To this end, we introduce InnerExpert, the first method to leverage these MoE-specific signals for per-token hallucination detection. InnerExpert combines routing-level and standard transformer signals into compact per-token feature vectors, classified by a lightweight detector trained on labels produced by an LLM-as-a-judge pipeline, which enables continuous model updates without manual annotation. Our results show that InnerExpert outperforms existing methods across five datasets and two MoE architectures, achieving up to 0.91 answer-level and 0.76 token-level AUROC, while requiring only a single forward pass.
