Search papers, labs, and topics across Lattice.
MemoAttack is introduced, a black-box jailbreak framework that systematically organizes and manages attack experience to improve LLM safety evaluation. It uses a skill-structured memory that pairs attack skills with templates, evidence, and lifecycle state, evolving this memory through evidence-based processes and balancing exploration/exploitation via contextual Thompson Sampling. Experiments on AdvBench show MemoAttack achieves a 98% attack success rate, outperforming the strongest baseline by 16.67% while reducing request count by 45.9%.
LLMs are now 16% more jailbreakable thanks to a novel attack strategy that evolves a structured memory of successful exploits.
Jailbreak attacks on large language models (LLMs) aim to induce LLMs to produce content that they are expected to refuse. Automated black-box jailbreak generation is especially important for safety evaluation, where the attacker observes only model outputs and needs to automatically search for effective adversarial prompts. Existing black-box jailbreak methods either depend on sample-wise heuristic search or leverage attack experience through accumulating strategy pools or method libraries, lacking a systematic organization and management of attack experience. To mitigate these drawbacks, we propose MemoAttack, a memory-driven black-box jailbreak framework with comprehensive attack memory modeling, evolution, and selection. Specifically, MemoAttack comprises three key designs: (1) Skill-Structured Memory Modeling, which abstracts accumulated attack experience into reusable skill-structured attack memory whose units pair attack skills with templates, evidence, and lifecycle state; (2) Lifecycle-Driven Memory Evolution, which evolves the memory through evidence-based probation, promotion, retirement, reactivation, elimination, and storage cleanup; and (3) Explore-Exploit Balanced Memory Selection, which balances reliable memory reuse with uncertainty-driven exploration via contextual Thompson Sampling. Experiments on AdvBench demonstrate that MemoAttack achieves an average attack success rate of 98.00%, outperforming the strongest baseline by 16.67 percentage points, while reducing request count by 45.9%. Moreover, MemoAttack continuously improves as memory accumulates over more samples.
