Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
To evaluate whether emergent agentic behaviors reflect genuine machine cognition, the authors evaluated 18 frontier language models against human baselines on AwarenessBench, a 14,381-sample testbed spanning metacognition, self-awareness, social awareness, and situational awareness across 15 sub-functions. While the top-performing model surpassed aggregate human baselines overall, most models exhibited severe performance drops in self-awareness and metacognition. Crucially, the authors find that cognitive awareness is orthogonal to standard language capabilities, meaning gains in raw reasoning performance do not reliably translate into improved introspective or situational awareness.
Scaling reasoning and pretraining loss does not buy self-awareness: top models beat human averages overall yet still fail markedly at metacognition, proving awareness is an independent capability axis.
As language models (LMs) exhibit increasingly consciousness-like behaviors, evaluating their cognitive abilities becomes essential. We introduce AwarenessBench, the first comprehensive benchmark for assessing the cognitive abilities of LMs in four dimensions: metacognition, self-awareness, social awareness, and situational awareness, covering 15 cognitive functions and 14,381 samples. Evaluating 18 state-of-the-art LMs, we find that all consistently surpass random baselines, with more advanced models performing better. We further compare LMs with human performance across three demographic groups, where the best-performing model surpasses human averages overall, but most still fall markedly short in metacognition and self-awareness. Finally, we show that awareness is a distinct capability: progress in language modeling or reasoning does not necessarily translate into improved cognition.