Search papers, labs, and topics across Lattice.
This paper critically examines the assumption that general intelligence is inherently valuable by exploring its evolutionary implications. It argues that many species have thrived for eons without general intelligence, while humans, despite their cognitive sophistication, face existential threats partly due to their intelligence. The authors propose that the risks associated with general intelligence may not stem from misalignment but from intrinsic structural features, raising ethical concerns about the development of AGI and artificial consciousness.
General intelligence may not only be overrated but could also pose unique existential risks that non-intelligent species avoid.
The pursuit of artificial general intelligence (AGI) rests on a seemingly self-evident premise: that general intelligence, the kind of flexible, domain-general cognitive capacity exemplified by Homo sapiens, is extraordinarily valuable. This paper subjects this premise to critical scrutiny. We first present the intuitive case for the value of general intelligence before mounting an evolutionary challenge. We argue that, on evolutionary timescales, its adaptive value is far from empirically established. Numerous taxa, from cyanobacteria to horseshoe crabs, have persisted for hundreds of millions or even billions of years without anything resembling general intelligence, while Homo sapiens has existed for roughly 300,000 years and already faces self-generated existential risks. Mass extinction events do not preferentially favour cognitively sophisticated species. We argue that general intelligence may be the only biological strategy that generates existential threats to the species possessing it, an existential risk paradox with no parallel among non-intelligent survival strategies. Unlike prevailing accounts of AI risk that trace the danger to misalignment, we locate it in structural features of general intelligence itself, implying that even well-aligned AGI would inherit this liability. If the long-term evolutionary value of general intelligence is uncertain or negative, this raises ethical questions about engineering AGI and, more urgently, creating artificial consciousness. Drawing on deontological ethics and the precautionary principle, we argue that this uncertainty imposes a duty of caution: if we create a new kind of intelligent being, we bear responsibility for ensuring the conditions under which it can flourish.
