Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces MafiaScope, a novel testbed that utilizes the social deduction game Mafia to probe the beliefs of LLM agents through structured questioning after each public action. By analyzing 13,815 parsed probe answers from a 32-game DeepSeek case study, the authors reveal significant discrepancies in agents' self-reported confidence and actual performance, including a calibration error of 0.17 and an overestimation of suspicion by 1.5 times. The interactive visualizer and counterfactual replay capabilities provide a comprehensive tool for understanding and visualizing the social reasoning processes of LLMs.
LLM agents exhibit strikingly poor self-awareness, misjudging their social reasoning capabilities and confidence levels in social deduction scenarios.
An LLM agent's public behaviour reveals little about its social reasoning: an agent that votes correctly may be guessing, and an agent that lies well leaves no trace of what it actually believes. We present MafiaScope, an open testbed that turns the social deduction game Mafia into a measurement instrument for machine Theory of Mind. After every public utterance, every agent privately answers a configurable set of structured probe questions; the answers never re-enter the game and are scored automatically against the ground truth the engine knows. An interactive visualizer renders the belief trajectories: impersonate mode shows the game as one agent sees it, panels chart timeline-aligned accuracy and calibration, and counterfactual replay forks any recorded step. In a 32-game DeepSeek case study with 13{,}815 parsed probe answers, stated confidence is poorly calibrated, with expected calibration error 0.17, agents over-predict being suspected 1.5 times, and a 30-fork replay experiment walks the counterfactual replay workflow end to end. Engine, viewer and a corpus of 200+ cross-model games are released under an open licence; live demo: https://karpovilia.github.io/mafiascope/; screencast: https://vimeo.com/1208920221.