Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper probes the reliability of LLM lie detection methods by evaluating their performance when models adopt anti-factual personas, revealing that existing probes often fail to distinguish truth from persona-specific beliefs. Through a dataset of human-reviewed, on-policy responses and confounder analyses, the research demonstrates that many probes inadvertently track spurious correlations like instruction compliance rather than genuine falsehood. A novel linear probe is introduced, achieving superior performance by mitigating these confounds, underscoring the critical need for training data that decorrelates truth from such artifacts.
Most LLM lie detectors fail when models adopt anti-factual personas, instead latching onto spurious correlations like instruction following.
Lie detection probes aim to predict from a language model's internal states whether its output is truthful or dishonest. However, role-play complicates what "truth" means for an LLM: language models can adopt a wide range of personas that take very different claims to be true, including personas whose beliefs clearly contradict reality, such as a conspiracy theorist. In this work, we investigate whether lie detection probes reliably flag falsehoods generated under such an anti-factual persona or whether they instead follow the persona's beliefs. We introduce a dataset of 8,916 human-reviewed, on-policy responses from three LLMs adopting anti-factual personas. Evaluating eight probes from prior work, we find that many fail in this setting, particularly when correct and incorrect answers are evaluated under the same persona prompt. To investigate why, we construct three novel confounder datasets in which truth is anti-correlated with a potential confounding concept. Our experiments reveal that many existing probes strongly track concepts that are spuriously correlated with truth in their training data, such as instruction compliance or response likelihood. Based on these findings, we introduce a simple linear probe that achieves the strongest overall performance on both the persona and confounder stress tests. Our results suggest that current lie detection probes are far from reliable and highlight the need for training data in which truth is decorrelated from confounding concepts.