Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study investigates the effectiveness of large language models (LLMs) as decision-making agents in online social network (OSN) simulations, comparing their performance to a finite-state machine reference policy. By evaluating three LLMs—LLaMA 3.1, GPT-OSS, and Mistral 24B—across various prompting strategies, the authors find that while LLMs can approximate the intended behavioral policy, they fail to do so consistently and often introduce biases. Notably, even the best-performing LLM configurations are significantly slower than traditional Markov chain sampling methods, highlighting the computational trade-offs involved in using LLMs for decision-making in simulations.
LLMs may approximate decision policies in social simulations, but their inconsistency and slow performance raise serious questions about their viability as replacements for traditional methods.
Large language models (LLMs) are increasingly used as decision-making components in social simulations. This introduces a methodological risk: the simulation may deviate from the explicit behavioral policy defined by the researcher. In online social network (OSN) simulations, action choices shape system dynamics, interaction patterns, and model interpretability. This paper evaluates whether LLM action selectors preserve an interpretable reference policy in an OSN simulation. The reference is a finite state machine implemented as a first-order Markov model, with transition probabilities depending on the user type. The evaluation uses a synthetic network with 1,000 agents and 10,000 action decisions. Three open-weight LLMs are tested: LLaMA 3.1, GPT-OSS, and Mistral 24B. Each model is evaluated under three prompting strategies: base, guided, and probabilistic. Alignment is measured using Jensen-Shannon Divergence with Laplace smoothing, and execution time is reported. Results show that LLMs can approximate the reference policy in some configurations, but do not preserve it reliably. Alignment varies across models and prompts, and additional guidance can introduce systematic action biases. Even the best-aligned LLM configurations are several hundred times slower than direct Markov chain sampling. These findings indicate that LLM-based action selection is not a direct replacement for explicit decision policies: it can alter the intended behavior while increasing computational cost.