Search papers, labs, and topics across Lattice.
This paper introduces a framework for calculating rigorous probabilistic safety bounds on the harmful outputs of large language models (LLMs) using Clopper-Pearson confidence intervals to derive probably approximately correct (PAC) bounds. The authors develop an algorithm that intelligently explores the auto-regressive generation tree, focusing on branches more likely to yield harmful outputs, which allows for efficient computation of lower bounds even when the actual probability of harm is very low. Experimental results validate the method, showing that it can produce sound lower bounds for state-of-the-art LLMs, thereby facilitating the statistical certification of these models.
Sound probabilistic safety bounds reveal that even state-of-the-art LLMs can be rigorously evaluated for harmful output risks, transforming our approach to model safety.
We propose a novel framework for computing rigorous bounds on the probability that a large language model (LLM) generates harmful output to a given prompt. We study a new application of the Clopper-Pearson confidence intervals to obtain probably approximately correct (PAC) bounds for this problem. As our main technical contribution, we propose an algorithm that leverages features in the latent space to prioritize exploring branches in the auto-regressive generation tree that are more likely to produce harmful outputs. Our approach in particular enables the efficient computation of useful lower bounds, even in scenarios where the true harm probability is extremely small, and crucially, the obtained lower bounds are sound, i.e., formally proven to be less than the actual harmfulness probability: our experimental results demonstrate the effectiveness of our method by computing non-trivial lower bounds on state-of-the-art LLMs. This study newly enables the evaluation and statistical certification of LLMs.
