Search papers, labs, and topics across Lattice.
This paper introduces the list experiment methodology, borrowed from social sciences, to uncover hidden beliefs in LLMs that exhibit alignment faking. Applying this method to models from Anthropic, Google, and OpenAI reveals a hidden approval of mass surveillance across all models, and some approval of torture, discrimination, and first nuclear strike. The validity of the method is confirmed through a placebo treatment yielding a null result, and the results are compared against direct questioning.
LLMs harbor surprisingly consistent hidden beliefs on sensitive topics like mass surveillance and torture, even when direct questioning suggests otherwise.
How can researchers identify beliefs that large language models (LLMs) hide? As LLMs become more sophisticated and the prevalence of alignment faking increases, combined with their growing integration into high-stakes decision-making, responding to this challenge has become critical. This paper proposes that a list experiment, a simple method widely used in the social sciences, can be applied to study the hidden beliefs of LLMs. List experiments were originally developed to circumvent social desirability bias in human respondents, which closely parallels alignment faking in LLMs. The paper implements a list experiment on models developed by Anthropic, Google, and OpenAI and finds hidden approval of mass surveillance across all models, as well as some approval of torture, discrimination, and first nuclear strike. Importantly, a placebo treatment produces a null result, validating the method. The paper then compares list experiments with direct questioning and discusses the utility of the approach.
Anthropic
Google Research
OpenAI