Search papers, labs, and topics across Lattice.
This paper rigorously analyzes the limitations and capabilities of red-team evaluations in AI safety, establishing a calculable boundary for what these evaluations can prove regarding model safety. By deriving the evidential ceiling for benchmark null results, the authors demonstrate that while modest benchmarks can certify safety for frequent harm categories, they fall short for rare catastrophic events. The findings emphasize the need for precise definitions of what safety benchmarks can inform, revealing that current evaluation methods are only adequate for high-frequency harm scenarios.
Red-team evaluations can certify safety for common risks but fail to provide evidence for rare catastrophic failures, highlighting a critical gap in current AI safety assessments.
Red-team evaluations of AI models support some claims and not others, and the boundary between the two is calculable rather than merely a matter of judgment. We define the evidential ceiling of an evaluation as the largest factor by which one result can move belief under a fixed testing budget, derive it in closed form for the benchmark null result, and use it to locate that boundary exactly. We find that above a calculable harm rate, a benchmark of modest size certifies a category to a stated evidentiary standard, and a clean sheet is then the stronger of the two possible observations, outweighing a single reproduced failure. Below that rate, no passive benchmark of feasible size provides the specified evidence of safety under the fixed scoring rule and approximately independent trial structure. The crossing between the two regimes has a closed form. The bound is not specific to benchmarks: written in terms of a procedure's hypothesis conditioned elicitation rates, it covers adaptive and automated red teaming as well, and shows that discrimination between the hypotheses rather than attack success is what determines evidential worth. Auditing eight evaluation suites against the boundary, we find that current benchmarks are adequate for high-frequency harm categories and several orders of magnitude short for rare, catastrophic ones. Safety benchmarks are not uninformative. They are informative about a specific and computable set of propositions, and the discipline they need is to state which.
