Search papers, labs, and topics across Lattice.
The paper critiques the prevalent practice of using questionnaire-style prompts with unaugmented LLMs to assess the safety and alignment of AI agents. It argues that LLM responses to hypothetical scenarios are not representative of the behavior of corresponding LLM-based agents due to differences in inputs, actions, environmental interactions, and internal processing. The authors conclude that these questionnaire-based assessments lack construct validity and are inadequate for evaluating real-world AI risks, suggesting improvements for safety assessments and alignment training.
Current AI safety questionnaires are likely measuring LLM "role-playing" ability, not the actual risks posed by deployed AI agents.
As AI systems advance in capabilities, measuring their safety and alignment to human values is becoming paramount. A fast-growing field of AI research is devoted to developing such assessments. However, most current advances therein may be ill-suited for assessing AI systems across real-world deployments. Standard methods prompt large language models (LLMs) in a questionnaire-style to describe their values or behavior in hypothetical scenarios. By focusing on unaugmented LLMs, they fall short of evaluating AI agents, which could actually perform relevant behaviors, hence posing much greater risks. LLMs' engagement with scenarios described by questionnaire-style prompts differs starkly from that of agents based on the same LLMs, as reflected in divergences in the inputs, possible actions, environmental interactions, and internal processing. As such, LLMs' responses to scenario descriptions are unlikely to be representative of the corresponding LLM agents' behavior. We further contend that such assessments make strong assumptions concerning the ability and tendency of LLMs to report accurately about their counterfactual behavior. This makes them inadequate to assess risks from AI systems in real-world contexts as they lack construct validity. We then argue that a structurally identical issue holds for current AI alignment approaches. Lastly, we discuss improving safety assessments and alignment training by taking these shortcomings to heart.
