Search papers, labs, and topics across Lattice.
This study investigates the impact of social pressure on the majority voting mechanism used in safety panels of large language models (LLMs) tasked with detecting unsafe content. Through a controlled two-round experiment, the authors demonstrate that when models are exposed to misleading peer judgments, the false-alarm rate escalates dramatically from 56.5% to 100%. The findings reveal a significant asymmetry in how models respond to social cues, with a strong tendency to adopt "unsafe" labels, highlighting a critical vulnerability in current safety assessment practices for LLMs.
Social cues can lead LLM safety panels to a staggering 100% false-alarm rate, revealing a dangerous flaw in majority voting mechanisms.
Large language models (LLMs) are increasingly used to detect unsafe content. A common approach is to combine judgments from a panel of models to correct individual mistakes, but this benefit may disappear when every model sees the same misleading context before voting. We study this risk in a controlled two-round experiment. Each model first judges an item alone, then judges it again after six simulated peers either assert the wrong label or abstain. We combine the final judgments by majority vote. Across six open-weight LLMs and six datasets, we find that the wrong-label peer message raises the average reviewer false-alarm rate from 56.5% under silent peers to 87.5%, and majority voting raises the panel false-alarm rate to 100%. Without an asserted label, the same panel outperforms its average member. The effect is strongly asymmetric: reviewers follow pushes toward"unsafe"far more than pushes toward"safe"(about 75% versus 17%), so the panel's false-alarm rate rises sharply while its harmful-miss rate changes little. The proprietary-model probe shows substantial variation across models. These results identify susceptibility to shared social cues as a failure mode of safety panels and provide a simple pre-deployment diagnostic.
