Search papers, labs, and topics across Lattice.
This paper introduces a novel evaluation framework, Non-Conversational Planning Theory of Mind (NCP-ToM), to assess the ability of Large Language Models (LLMs) to induce belief states in other agents through actions rather than conversation. By testing six advanced models, including GPT-5 and Gemini 2.5 Pro, across 600 task instances, the study reveals that GPT-5 successfully achieved approximately 80% of the goals in an agentic setting, outperforming human participants while still demonstrating lower robustness across varied contexts. The results indicate that LLMs exhibit emerging social-reasoning capabilities, particularly in tasks that require inducing true belief states, which is crucial for aligning autonomous agents with human intentions.
GPT-5 outperformed humans in inducing belief states through action, highlighting a significant leap in LLMs' social reasoning capabilities.
Theory of Mind (ToM) benchmarks for Large Language Models (LLMs) typically rely on passive question-answering formats, but the deployment of LLMs in increasingly agentic and autonomous forms demands new evaluations. In this paper we evaluate an agent's ability to induce specific belief states in other agents by taking actions rather than using conversational persuasion, a capability we call Non-Conversational Planning ToM (NCP-ToM). NCP-ToM is likely to be essential for many agent use-cases, including within user-assistant interactions and pedagogical contexts, but may also present manipulation or misinformation risks. Using a novel framework, NCP-ExploreToM, we subvert the conventional task structure by providing models with a set of belief state goals and requiring them to move objects or direct characters into rooms to achieve their goals. We evaluated six frontier models, including GPT-5, Gemini 2.5 Pro and the Claude 4 series, and a cohort of human participants, across 600 task instances. GPT-5 was successful on approximately 80% of tasks in the agentic setting, and was the only model to outperform human participants on our task, but was still less robust than humans across contexts. We additionally found that all models, like humans, performed better on tasks inducing true belief states than false belief states, which is a positive signal for alignment efforts. These findings highlight emerging social-reasoning capabilities in LLMs for non-conversational task completion and underscore the necessity of agentic evaluations for understanding the safety and alignment of autonomous social agents.
