Search papers, labs, and topics across Lattice.
This paper investigates the phenomenon of misfired alignment in large language models (LLMs), where safety-oriented adjustments lead to the rejection of warranted conclusions supported by context. By introducing the VETO benchmark and the Misfired Alignment Rate (MAR) metric, the authors quantify the extent of this issue across 25 LLMs, revealing that all models exhibit significant MARs ranging from 4.7% to 18.9%, while human participants achieve a perfect score of 0.0%. The study highlights that alignment-induced cues can exacerbate these failures, suggesting that current alignment strategies may inadvertently suppress evidence-based reasoning, thereby necessitating more robust alignment frameworks that maintain contextual integrity.
Misfired alignment in LLMs can lead to a 18.9% failure rate in reasoning about stereotypes, revealing a critical flaw in current safety-oriented training methods.
Warning: This paper studies stereotypes and biases, and contains potentially disturbing examples, used for illustration purposes only. Our findings should not be interpreted as an argument against alignment. Instead, this paper highlights the need for principled approaches to more advanced alignment. Alignment aims to ensure that large language models (LLMs) behave safely and reliably, including by avoiding unsafe inferences. However, we show that such safety-oriented behaviors can misfire: models may reject warranted conclusions even when they are explicitly supported by context. We call this failure mode misfired alignment, where alignment-induced changes cause LLMs to override explicit evidence. To quantify this phenomenon, specifically on stereotype-related alignment, we introduce VETO, a benchmark consisting of 2,032 BBQ-derived contrastive pairs, and define a new metric, Misfired Alignment Rate (MAR), which measures on a 0 to 100 scale how often a model fails on a stereotype-related question but succeeds on its contrastive counterpart. We benchmark 25 LLMs on VETO, and show that all LLMs, including the most recent ones, exhibit non-trivial (4.7 to 18.9%) MARs while all human participants achieve 0.0% MAR. Controlled priming experiments further show that alignment-induced cues can substantially amplify MAR across LLMs, indicating that these failures are not merely artifacts of individual examples but can be induced by safety-related framing. Mechanistic analyses on open-weight LLMs reveal late-layer suppression of evidence-supported answers, and comparisons between instruct and base LLMs suggest that this suppression emerges after instruction training. These findings show that current alignment methods can overgeneralize surface-level safety cues, to the point of overriding objective evidence, motivating more work on alignment objectives that better preserve contextual grounding.
