Search papers, labs, and topics across Lattice.
This study evaluates the performance of nine open-weight large language models (LLMs) as agents in a networked Prisoner's Dilemma experiment, comparing their behavior to that of human participants. While the LLMs successfully replicate macro-level cooperation dynamics, such as the early decline and later stabilization of cooperation, they fail to capture the individual-level heterogeneity and nuanced decision-making patterns seen in humans. The research highlights a critical dissociation between collective outcomes and individual behaviors in LLMs, suggesting that outcome-level agreement is insufficient for validating LLMs as proxies for human decision-making.
LLMs can mimic human-like cooperation in aggregate, yet they fundamentally misrepresent individual decision-making dynamics.
Large language models (LLMs) are increasingly used as agents in simulations of social systems, yet it remains unclear when their behavior can be interpreted as a faithful proxy for human decision-making. Here we test LLM agents against a direct empirical benchmark: a large-scale networked Prisoner's Dilemma experiment with human participants. Using the same interaction protocol, payoff structure, and network topologies, we compare nine open-weight LLMs with the human data. The selected model reproduces several macro-level features of cooperation dynamics, including the early decline and later stabilization of cooperation. This aggregate agreement, however, does not extend uniformly to finer levels of behavior. LLM populations underestimate individual-level heterogeneity and generate conditional cooperation patterns that differ from those observed in humans. Adding a fraction of random agents improves some aspects of micro-level agreement, but does not remove the mismatch in decision rules. These findings reveal a macro--micro dissociation in LLM-based social agents: collective outcomes can appear human-like even when the underlying behavioral distributions and mechanisms are not. They suggest that validating LLM agents as human surrogates requires comparisons across aggregate dynamics, individual heterogeneity, and context-dependent decision rules, rather than outcome-level agreement alone.
