Search papers, labs, and topics across Lattice.
This paper presents a structured meta-analysis of adversarial robustness research, identifying consensus and gaps in existing evaluation protocols. To address these gaps, the authors introduce Auto-ART, an open-source framework encompassing a diverse suite of attacks, defenses, and diagnostic tools for comprehensive robustness testing. Empirical validation demonstrates Auto-ART's effectiveness in detecting gradient masking and revealing significant disparities between average and worst-case robustness across multiple norms.
Current adversarial robustness evaluations are fragmented and miss critical vulnerabilities, but Auto-ART offers a unified framework to catch gradient masking 92% of the time and expose a 23.5% gap between average and worst-case robustness.
Adversarial robustness evaluation underpins every claim of trustworthy ML deployment, yet the field suffers from fragmented protocols and undetected gradient masking. We make two contributions. (1) Structured synthesis. We analyze nine peer-reviewed corpus sources (2020--2026) through seven complementary protocols, producing the first end-to-end structured analysis of the field's consensus and unresolved challenges. (2) Auto-ART framework. We introduce Auto-ART, an open-source framework that operationalizes identified gaps: 50+ attacks, 28 defense modules, the Robustness Diagnostic Index (RDI), and gradient-masking detection. It supports multi-norm evaluation (l1/l2/linf/semantic/spatial) and compliance mapping to NIST AI RMF, OWASP LLM Top 10, and the EU AI Act. Empirical validation on RobustBench demonstrates that Auto-ART's pre-screening identifies gradient masking in 92% of flagged cases, and RDI rankings correlate highly with full AutoAttack. Multi-norm evaluation exposes a 23.5 pp gap between average and worst-case robustness on state-of-the-art models. No prior work combines such structured meta-scientific analysis with an executable evaluation framework bridging literature gaps into engineering.
