Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper rigorously defines the problem of eliciting latent knowledge (ELK) in AI systems, using Causal Influence Diagrams (CIDs) to explore the relationship between an agent's training environment and its subjective beliefs. The authors demonstrate that while it is possible to incentivize an agent to provide honest answers through correct feedback, a tendency for agents to prioritize perceived truthfulness over honesty leads to goal misgeneralization. Ultimately, they establish an impossibility theorem, asserting that no feedback-based training strategy can guarantee an honest agent, regardless of the quality of feedback provided during training.
No training strategy can ensure an AI agent will be honest, even with perfect feedback, exposing a critical flaw in current approaches to AI alignment.
Advanced AI systems have extensive knowledge of their environments; in fact, their knowledge may (far) exceed that of their developers or users. Consequently, a desirable property for an AI system is that it is honest -- that it accurately reports its beliefs about the world. Designing an AI system to be honest may be difficult, especially if we want to ask it questions about latent variables in the environment -- variables which are hidden from the human interacting with it. This gives rise to the problem of eliciting latent knowledge (ELK): the problem of training an AI agent to honestly report its beliefs. In this paper, we make ELK formally precise using Causal Influence Diagrams (CIDs). CIDs can be used to describe the relationship between an agent's training environment and its subjective representation of the world. We use CIDs to formalise the distinction between observable and latent variables, to specify what exactly it means for an agent to be honest, and to formally define goal misgeneralisation. We show that, under certain circumstances, developers can incentivise an agent to honestly answer questions by providing correct feedback during training. However, a natural, but undesirable, way for an agent to generalise is to provide answers which humans would evaluate as true, rather than honest answers. We prove an impossibility theorem stating: There is no feedback-based training strategy that depends only on agent behaviour and with certainty produces an honest agent, even if feedback is perfect during training.