Search papers, labs, and topics across Lattice.
This paper evaluates the reasoning capabilities of Gemini 2.5 Pro and Flash, Llama 3.3 70B, and GPT-OSS 120B on General Game Playing (GGP) tasks, assessing their ability to perform forward simulation, legal action generation, and multistep state formulation. The authors correlate LLM performance with 40 structural features of GGP games and analyze the impact of game obfuscation on performance. Results show strong performance across models, with degradation over longer horizons, and identify common reasoning errors like rule hallucination and syntactic errors.
LLMs can play games, but their reasoning falters as the game progresses, revealing vulnerabilities to hallucinated rules and redundant state facts.
This paper examines the reasoning capabilities of Large Language Models (LLMs) from a novel perspective, focusing on their ability to operate within formally specified, rule-governed environments. We evaluate four LLMs (Gemini 2.5 Pro and Flash variants, Llama 3.3 70B and GPT-OSS 120B) on a suite of forward-simulation tasks-including next / multistep state formulation, and legal action generation-across a diverse set of reasoning problems illustrated through General Game Playing (GGP) game instances. Beyond reporting instance-level performance, we characterize games based on 40 structural features and analyze correlations between these features and LLM performance. Furthermore, we investigate the effects of various game obfuscations to assess the role of linguistic semantics in game definitions and the impact of potential prior exposure of LLMs to specific games during training. The main results indicate that three of the evaluated models generally perform well across most experimental settings, with performance degradation observed as the evaluation horizon increases (i.e., with a higher number of game steps). Detailed case-based analysis of the LLM performance provides novel insights into common reasoning errors in the considered logic-based problem formulation, including hallucinated rules, redundant state facts, or syntactic errors. Overall, the paper reports clear progress in formal reasoning capabilities of contemporary models.
