Search papers, labs, and topics across Lattice.
This paper introduces the Threshold Exceedance Criteria (TEC) framework to evaluate the potential uplift in CBRN planning capabilities when non-expert actors access frontier language models compared to public tools. Through a large-scale empirical study, the authors assessed both generative and revisionist uplift in CBRN domains, revealing that while model-assisted plans sometimes achieved expert-equivalent ratings, significant material uplift was predominantly observed only in the radiological domain. The findings provide critical insights for policymakers and model developers, emphasizing the need for robust evaluation methodologies in assessing risks associated with advanced language models.
Model-assisted CBRN planning shows promise, but significant uplift is limited to the radiological domain, challenging assumptions about AI's broad applicability in high-stakes scenarios.
As frontier language models advance, policymakers and model developers need methods for assessing whether model access materially increases a non-expert actor's ability to plan high-consequence Chemical, Biological, Radiological, or Nuclear (CBRN) misuse relative to public tools alone. Existing CBRN evaluations differ in non-expert definitions, threat scope, baselines, scoring rubrics, and decision rules, making results difficult to compare across studies. We introduce a Threshold Exceedance Criteria (TEC) framework that decomposes an uplift study into independently executable components: determining non-expert participant eligibility, defining the CBRN threat scope for the study, and statistically estimating material uplift. We then operationalize the TEC framework in a large-scale empirical study using a design that determines two forms of uplift: generative (where a model assists plan creation from scratch) and revisionist (where a model assists refinement of an existing plan). The study produced attack plans across the CBRN domains, which we evaluated through subject-matter-expert review to estimate generative and revisionist uplift. Applying the framework, our empirical study revealed domain heterogeneity: under this controlled pre-release evaluation, model-assisted plans sometimes received expert-equivalent instructional ratings, but confirmed material uplift was limited to the radiological domain. These findings informed mitigation and deployment-governance decisions rather than characterizing deployed model behavior. We conclude with methodological lessons for future CBRN uplift evaluations, emphasizing prespecified criteria, explicit baselines, separation of generative and revisionist estimates, and careful distinction between preliminary screening signals and confirmed risk determinations.
