Search papers, labs, and topics across Lattice.
This paper investigates the stability of evaluators in automated LLM vulnerability scanners, revealing that Attack Success Rates (ASR) are highly sensitive to the choice of evaluator, even when the attacks and model outputs remain constant. To address this, they propose a two-phase, reliability-aware evaluation framework that first quantifies evaluator disagreement and then uses a verification-based evaluation method to validate evaluators. Applied to the Garak scanner, the framework significantly improves evaluator accuracy and reduces uncertainty in ASR estimates, demonstrating the importance of accounting for evaluator instability in LLM security assessments.
LLM vulnerability scanners can give you wildly different security scores depending on *who* is judging the attacks, not just the model's weaknesses.
Automated LLM vulnerability scanners are increasingly used to assess security risks by measuring different attack type success rates (ASR). Yet the validity of these measurements hinges on an often-overlooked component: the evaluator who determines whether an attack has succeeded. In this study, we demonstrate that commonly used open-source scanners exhibit measurement instability that depends on the evaluator component. Consequently, changing the evaluator while keeping the attacks and model outputs constant can significantly alter the reported ASR. To tackle this problem, we present a two-phase, reliability-aware evaluation framework. In the first phase, we quantify evaluator disagreement to identify attack categories where ASR reliability cannot be assumed. In the second phase, we propose a verification-based evaluation method where evaluators are validated by an independent verifier, enabling reliability assessment without relying on extensive human annotation. Applied to the widely used Garak scanner, we observe that 22 of 25 attack categories exhibit evaluator instability, reflected in high disagreement among evaluators. Our approach raises evaluator accuracy from 72% to 89% while enabling selective deployment to control cost and computational overhead. We further quantify evaluator uncertainty in ASR estimates, showing that reported vulnerability scores can vary by up to 33% depending on the evaluator. Our results indicate that the outputs of vulnerability scanners are highly sensitive to the choice of evaluators. Our framework offers a practical approach to quantify unreliable evaluations and enhance the reliability of measurements in automated LLM security assessments.
