Search papers, labs, and topics across Lattice.
To disentangle stable model preferences from prompt-wording artifacts, this work defines a "bias depth score" that measures whether an LLM's response concentration persists across semantic scenario reframings across 4,442 opinion prompts. Testing on four models reveals that only ~25% of concentrated choices survive reframing as persistent "Deep biases," while the rest are prompt-dependent "Shallow biases." Critically, deep biases originate in pretraining, survive supervised fine-tuning, and prove significantly more resistant to prompt-based and fine-tuning debiasing strategies than shallow artifacts.
Nearly three-quarters of what benchmarks flag as model bias is just prompt-wording noise, but the surviving 25% are deeply entrenched pretraining representations that stubbornly resist post-training alignment.
Large language models often repeatedly select the same answer even when many alternatives are plausible. Prior work treats this concentration as bias, but it does not distinguish stable model preferences from responses that depend on a particular prompt wording. We introduce a bias depth score that measures both how strongly a model prefers its top answer under direct prompting and whether that answer survives scenario reframing. Across 4,442 opinion prompts and four large language models, only about a quarter of the concentrated preferences survive reframing. We call these persistent cases Deep biases, and the remaining prompt-dependent cases Shallow biases. Our results show that Deep biases are more often inherited from pretraining and preserved through SFT. Under both continued fine-tuning and prompt-based debiasing for diversity, Deep biases are consistently harder to remove than Shallow biases. Bias depth therefore separates stable learned biases from prompt-wording artifacts that single-prompt metrics conflate. Code, models, and data are available at deepbias.github.io.
