Search papers, labs, and topics across Lattice.
This study investigates how alignment tuning in large language models (LLMs) influences their susceptibility to biases such as sycophancy, revealing that these biases are primarily introduced during the alignment phase rather than pretraining. By analyzing hidden states across five model families and seven bias types, the authors demonstrate that biases manifest as distinct, coherent directions in the model's representation space, allowing for targeted debiasing interventions. The findings highlight that while biases are representationally distinct, they can be effectively managed without sacrificing overall model performance.
Alignment tuning installs distinct bias directions in LLMs, allowing for targeted debiasing that recovers unbiased answers while maintaining performance.
Modern LLMs are alarmingly susceptible to surprisingly simple immaterial changes of input prompts: a casual hint, an incorrectly labeled few-shot example, or a fake prior assistant turn often flips an originally correct answer. We study where this susceptibility, spanning sycophancy and related cue-induced biases, lives inside the model. Across five model families and seven BCT bias types, we extract a per-bias direction from hidden states and triangulate it through three measures: probing, leave-one-dataset-out transfer, and causal intervention. The susceptibility is largely installed by alignment tuning rather than pretraining: pretrained base models barely cave to these biases, and their activations carry no cue-specific signal beyond question content. Within aligned models, each bias becomes a single coherent direction that we can both decode and steer along, recovering the unbiased answer across every family we test. The biases stay representationally distinct, however: cross-bias entanglement is model-specific rather than a property of the bias category, and even behaviorally similar biases occupy different directions. The same intervention also serves as a modest debiasing tool, recovering a meaningful share of bias-induced errors while preserving most correct answers across all instruct families. Cue-induced bias is therefore best understood not as a single flaw in LLMs but as a family of distinct, causally active directions that alignment tuning installs.
