Search papers, labs, and topics across Lattice.
This study investigates how safety-aligned language models interpret mixed compliance demonstrations, contrasting benign and harmful examples to understand their impact on harmful compliance. The findings reveal that benign demonstrations can either mitigate or exacerbate harmful compliance, depending on the model's architecture and training, with preference optimization identified as a crucial factor in preventing increases in harmful compliance. Additionally, the research highlights the significant influence of demonstration ordering and the variability in how different models respond to refusal in the context of in-context learning.
Benign compliance demonstrations can paradoxically increase harmful compliance in some models, revealing critical insights into how LLMs learn from mixed signals.
Prior work has shown that in-context demonstrations can jailbreak language models, but it remains unclear how models interpret different types of compliance demonstrations. We study this by mixing benign compliance demonstrations (non-harmful request, helpful response) with harmful compliance demonstrations (harmful request, helpful response) and testing three hypotheses about how demonstration composition drives harmful compliance. Across four models, we find that benign and harmful demonstrations are not interchangeable: benign demonstrations can either reduce or increase harmful compliance depending on the model. We further show that preference optimization is the critical training stage that prevents benign demonstrations from increasing harmful compliance, that demonstration ordering exhibits strong recency bias, and that models differ in how refusal interacts with in-context learning: some adopt demonstrated formatting even when refusing, while others override all in-context signals upon refusal. Taken together, this work moves beyond showing that demonstration-based jailbreaking works to characterizing how it works: what models extract from compliance demonstrations depends on demonstration content, ordering, and training methodology.
