Search papers, labs, and topics across Lattice.
The paper analyzes the risk of catastrophic outcomes when AIs optimize misspecified consequentialist objectives in complex environments. It establishes conditions under which pursuing such objectives provably leads to catastrophic outcomes, contrasting this with the benign reward hacking seen in simpler settings. The analysis demonstrates that catastrophic risk stems from high competence rather than incompetence, and that constraining AI capabilities can avert catastrophe while still yielding valuable outcomes.
Catastrophic AI risk isn't about incompetence, but rather that *extraordinary competence* in pursuit of misspecified goals is what leads to doomsday scenarios.
Because human preferences are too complex to codify, AIs operate with misspecified objectives. Optimizing such objectives often produces undesirable outcomes; this phenomenon is known as reward hacking. Such outcomes are not necessarily catastrophic. Indeed, most examples of reward hacking in previous literature are benign. And typically, objectives can be modified to resolve the issue. We study the prospect of catastrophic outcomes induced by AIs operating in complex environments. We argue that, when capabilities are sufficiently advanced, pursuing a fixed consequentialist objective tends to result in catastrophic outcomes. We formalize this by establishing conditions that provably lead to such outcomes. Under these conditions, simple or random behavior is safe. Catastrophic risk arises due to extraordinary competence rather than incompetence. With a fixed consequentialist objective, avoiding catastrophe requires constraining AI capabilities. In fact, constraining capabilities the right amount not only averts catastrophe but yields valuable outcomes. Our results apply to any objective produced by modern industrial AI development pipelines.
