Christian Nickel

Bonn-Aachen International Center for Information Technology (b-it), University of Bonn, Lamarr Institute for Machine Learning and Artificial Intelligence, Research Center Trustworthy Data Science and Security (RC-Trust), University of Duisburg-Essen Abstract Theory of Mind (ToM) refers to an agent’s ability to model the internal states of others. Contributing to the debate whether large language models (LLMs) exhibit genuine ToM capabilities, our study investigates their ToM robustness using perturbations on false-belief tasks and examines the potential of Chain-of-Thought prompting (CoT) to enhance performance and explain the LLM’s decision. We introduce a handcrafted, richly annotated ToM dataset, including classic and perturbed false belief tasks, the corresponding spaces of valid reasoning chains for correct task completion, subsequent reasoning faithfulness, task solutions, and propose metrics to evaluate reasoning chain correctness and to what extent final answers are faithful to reasoning traces of the generated CoT. We show a steep drop in ToM capabilities under task perturbation for all evaluated LLMs, questioning the notion of any robust form of ToM being present. While CoT prompting improves the ToM performance overall in a faithful manner, it surprisingly degrades accuracy for some perturbation classes, indicating that selective application is necessary. 1 Introduction Theory of Mind (ToM) refers to an agent’s ability to infer and track the beliefs, intentions, and emotions of others (Premack and Woodruff, 1978; Rabinowitz et al., 2018; Kosinski, 2023). This ability to model the mental states of others is fundamental in human cognition and social interaction (Premack and Woodruff, 1978). Hence, enabling reliable ToM abilities in AI agents could unlock a range of new applications involving human-AI interactions, e.g. in assistive healthcare (Cuzzolin et al., 2020; Langley et al., 2022), empathetic conversational agents (Wang et al., 2024), education (Asthana and Collins-Thompson, 2024) or expert support, and cyber-physical systems like autonomous driving (Montese, 2024). Human ToM has been studied intensely in psychology and neuroscience, but the evidence of ToM in Large Language Models (LLMs) is mixed. While promising results have been reported initially (Kosinski, 2023), the underlying mechanisms remain unclear (Ullman, 2023): Are LLMs truly reasoning about mental states, or merely leveraging statistical regularities? Prior claims of ToM in LLMs often rely on narrow benchmarks that fail under small perturbations, calling into question their generality and interpretability. Existing benchmarks lack a systematic structure for evaluating these effects, and do not provide the means to isolate the impact of specific perturbation types or prompting strategies. S1. The non-transparent bag contains sweets. →label\xrightarrow{\ \text{\footnotesize label}\ } unknown