Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces UNIATTACK, an adversarial testing framework that systematically constructs effective black-box attack prompts against large language models (LLMs) by extracting and optimizing minimal high-impact features from existing attacks. UNIATTACK significantly outperforms prior methods, achieving an average attack success rate improvement of 64.63% to 248.82% against models with multi-layered defense mechanisms while maintaining a fraction of the cost. This innovative approach enables one-shot attacks that generalize across various models and safety categories, enhancing the assessment of LLM robustness.
UNIATTACK can breach multi-layered defenses with unprecedented efficiency, achieving up to 248% higher attack success rates than existing methods.
Large language models (LLMs) have demonstrated remarkable capabilities across a wide range of tasks. However, their safety remains a critical concern due to their susceptibility to adversarial prompt-based attacks. In this paper, we present UNIATTACK, an adversarial testing framework designed from a defense-oriented perspective to systematically construct effective black-box attack prompts. Unlike prior approaches that rely on static templates or iterative model-specific tuning, UNIATTACK extracts minimal but high-impact attack features from diverse existing attacks, optimizes them via a specialized attacker LLM, and composes them into flexible templates through automated refinement process. This feature-centric construction enables one-shot attacks that generalize across multiple models and safety categories, providing a practical tool for assessing LLM robustness. Our evaluation results shows that compared to the baselines, UNIATTACK achieves an average attack success rate (ASR) improvement of 64.63\%-248.82\% on models deployed with multi-layered defense mechanisms and it only takes 0.03\%-4.96\% cost of the baselines. UNIATTACK artifact is available at https://anonymous.4open.science/r/UniAttack-Artifact-30F1.