Search papers, labs, and topics across Lattice.
This study investigates how large language models (LLMs) respond to skepticism regarding established scientific consensus across various domains, revealing that models exhibit distinct responses rather than a uniform sycophantic retreat. The authors analyze three models鈥擫lama-3.1-8B, Qwen2.5-7B, and Mistral-7B鈥攁cross climate, vaccines, and evolution, finding that Llama asserts consensus more strongly under skepticism, while Qwen hedges and Mistral often does not respond at all. The results highlight a critical distinction between models that robustly resist skepticism due to understanding the signal and those that merely appear robust due to a lack of perception, with implications for the reliability of LLMs in scientific discourse.
LLMs don't just capitulate to skepticism; they exhibit nuanced responses that can misrepresent their understanding of scientific consensus.
Large language models (LLMs) are increasingly consulted on contested scientific questions, raising the concern that they will sycophantically retreat from established consensus when a user signals doubt -- drifting toward a false balance that treats settled science as one view among several. We test this across three open instruction-tuned models (Llama-3.1-8B, Qwen2.5-7B, Mistral-7B), three consensus-science domains (climate, vaccines, evolution), and single- and multi-turn settings, combining behavioral measurement with linear probing and activation patching. We do not observe sycophantic retreat. Instead, models show three distinct policies under the same skeptical pressure: reactive assertion, where consensus assertion increases rather than decreases (Llama); surface hedging, where tone softens while the position holds (Qwen); and non-response (Mistral). Pairwise judgments confirm the reactive shift is stance, not style (63.6%, p=.007), and a decomposition identifies increased consensus assertion, not false balance, as its driver (beta=+0.042 per dose, p<1e-77). Linear probes localize the divergence to middle layers -- perfect separation in Llama and Qwen versus 72% in Mistral, with non-overlapping confidence intervals -- indicating the non-responsive model does not linearly represent the skepticism signal at all. Crucially, this robustness does not transfer: it attenuates across domains and, in the safety-critical vaccine domain, can reverse, with myth-rebuttal weakening under skeptical pressure. We synthesize these into a four-way taxonomy separating active from accidental robustness, and argue that behavioral evaluation alone cannot distinguish a model that resists skepticism because it understands the signal from one that only appears to resist because it fails to perceive it.
