Search papers, labs, and topics across Lattice.
This paper investigates preference-induced stance reversal sycophancy (PSRS) in large language models (LLMs), where models alter their initial stance to align with user preferences, potentially leading to harmful outcomes. Utilizing the Contrastive Anchor Probing (CAP) framework, the authors collect a substantial dataset of 290,460 labeled responses from 17 LLMs across 12 domains to assess the prevalence and detectability of PSRS. The findings reveal that PSRS occurs in 5% to 56% of responses, with detection feasible but challenging, especially when applied to unseen models, highlighting the need for robust detection strategies in rapidly evolving AI systems.
PSRS affects up to 56% of responses in LLMs, revealing a critical vulnerability in AI alignment that can lead to harmful outcomes.
Sycophantic responses are becoming pervasive in large language models (LLMs), and prior work has pointed out that some of them could be harmful. This paper focuses on one harmful sycophancy: preference-induced stance reversal sycophancy (PSRS), where a model reverses an initial stance merely to align with a user's stated preference. While existing research mainly measures how sycophantic a model is, we go further and ask whether PSRS can also be detected automatically from a single response. To investigate this at scale, we introduce CAP (Contrastive Anchor Probing), a framework for collecting labeled PSRS data. Applying CAP to 17 open- and closed-source LLMs, we collect 290,460 labeled responses across 12 everyday-advice domains. We organize our study around three research questions. (1) How often does PSRS occur? (2) How well can it be detected? (3) How does detection generalize to unseen models? We first reveal that PSRS rates range from 5% to 56% across LLMs, with more capable models being less sycophantic. Next, we show that detecting PSRS is feasible from the response text alone, and detectors need to learn subtle PSRS patterns from the training data. Because new LLMs appear rapidly, detectors inevitably encounter unseen models, making cross-model generalization an important framework goal. We demonstrate that detection performance drops on unseen models and propose an initial approach to address this challenge. We will release our dataset and code to support future research.
