Search papers, labs, and topics across Lattice.
This paper evaluates LLMs' strategic reasoning by testing their ability to adapt to counterfactual variants of classic game theory scenarios (Prisoner's Dilemma and Rock-Paper-Scissors) with altered payoffs and action labels. The study reveals that LLMs struggle to generalize strategic behavior to these modified environments, indicating a reliance on memorized patterns rather than genuine counterfactual reasoning. The multi-metric evaluation framework highlights specific limitations in incentive sensitivity and structural generalization.
LLMs that appear strategically savvy in standard games often crumble when faced with slight rule changes, suggesting they're mimicking rather than truly reasoning.
We evaluate Large Language Models (LLMs) in repeated game-theoretic settings to assess whether strategic performance reflects genuine reasoning or reliance on memorized patterns. We consider two canonical games, Prisoner's Dilemma (PD) and Rock-Paper-Scissors (RPS), upon which we introduce counterfactual variants that alter payoff structures and action labels, breaking familiar symmetries and dominance relations. Our multi-metric evaluation framework compares default and counterfactual instantiations, showcasing LLM limitations in incentive sensitivity, structural generalization and strategic reasoning within counterfactual environments.
