Search papers, labs, and topics across Lattice.
This paper introduces VanillaBench, a benchmark that quantifies the accuracy gap between adversarially-trained models and their vanilla counterparts across multiple threat models. By analyzing 186 robust models catalogued by RobustBench, the authors reveal that the mean clean accuracy delta relative to the best vanilla model ranges from -7.7 to -29.5 percentage points, highlighting a significant trade-off between robustness and standard accuracy. The findings underscore the necessity for future robustness evaluations to include vanilla-referenced accuracy gaps, as this information is vital for informed decision-making in real-world applications.
Adversarially-trained models may sacrifice up to 29.5 percentage points in standard accuracy compared to their vanilla counterparts, a gap that has been largely overlooked in existing research.
Adversarial robustness research has produced hundreds of defended models over the past decade, yet the literature almost universally reports robustness results in isolation: standard (clean) accuracy and adversarial accuracy of the robust model are shown, but the gap to the corresponding vanilla model is rarely quantified. We introduce VanillaBench, a systematic benchmark that makes this gap explicit. For every adversarially-trained model catalogued by RobustBench across four threat models, we compute the accuracy difference against multiple vanilla references from Papers with Code, computed over both all entries and no-extra-data entries, the best vanilla model as of the robust model's publication year, and an architecture-matched baseline. Across all 186 robust models, the mean delta clean relative to the best vanilla model ranges from -7.7 to -29.5 percentage points, and even the single most robust model per track still trails its temporal vanilla counterpart by 4.0-21.0 points. The architecture-matched comparison, which isolates the effect of adversarial training from architectural differences, reveals a mean gap of -3.5 to -17.5 points. Restricting this architecture-matched comparison to models whose vanilla accuracy is known for the exact same architecture, rather than approximated from a related one, narrows the gap to -4.0 to -14.0 points. These results demonstrate that the robustness-accuracy trade-off is substantially larger than what is typically conveyed by individual papers. This information is critical for practitioners and decision-makers. When deploying models in real-world settings, the accuracy cost of robustness directly affects business outcomes, yet current publications do not provide the vanilla baseline needed to assess it. We argue that future robustness evaluations should report vanilla-referenced accuracy gaps as a standard component.
