Search papers, labs, and topics across Lattice.
This paper explores how perturbation techniques can transform ineffective jailbreak prompts into effective ones, posing a significant security risk to large language model (LLM) safety. By analyzing the internal representations of perturbed inputs in various Qwen and Llama models, the authors identify distinct patterns in the last-layer-last-token embedding space and the next-token probability space. Notably, they find that while the former space categorizes prompts based on their structure, the latter reveals a complex clustering behavior, with only a few specific tokens showing a strong correlation with compliant responses.
Perturbation techniques can exploit subtle internal representations in LLMs, revealing vulnerabilities that could compromise model safety.
Perturbation techniques that turn unsuccessful jailbreak prompts into successful ones are continuously evolving, constituting a major security threat to LLM safety. In this paper, we investigate the internal representations of such string-level perturbed jailbreak inputs in the small weight models of the Qwen-2.5-1.5B/-3B/-7B-Instruct and Llama-3.2-1B/-3B/-3.1-8B-Instruct families. We select two representation spaces: the last-layer-last-token embedding space and the top-50 next-token probability space. The former space separates prompts based on their spelling and format, while the latter space is effectively one-dimensional but appears more complex to cluster. Within our refusal-dominated answer set we find no behavioral hyperplane in either space. Only the next token "Sure" in the 1.5B Qwen model, and both tokens "," and "膴膴" in the 1$ Llama model, display a significant association with a compliant-labeled answer.
