Search papers, labs, and topics across Lattice.
This study investigates the impact of scaling on the fidelity of social simulations using Large Language Models (LLMs) across three domains: opinion modeling, behavioral simulation, and longitudinal forecasting. By analyzing a suite of 85 transformer LLMs with varying compute budgets, the authors find that scaling generally enhances performance in behavioral and opinion tasks, particularly for well-represented populations, while revealing that certain tasks, such as longitudinal forecasting and those involving underrepresented opinions, show slower improvements. The research highlights that while scaling benefits most social simulation tasks, it does not uniformly enhance model calibration for human cognitive biases and heuristics, indicating a nuanced relationship between model size and simulation fidelity.
Scaling LLMs significantly boosts social simulation accuracy in well-represented domains, but fails to enhance calibration for human cognitive biases.
Large Language Model (LLM) social simulations are a promising research method, but they are not yet faithful enough to be adopted widely. In this work, we investigate whether the current scaling paradigm in language modeling is likely to close these gaps, or whether simulation fidelity is orthogonal to general capabilities and therefore deserving of more research attention. We use scaling laws to study the relationship between LLMs'compute scale, general capability benchmarks, and the fidelity of social simulation in three representative sub-domains: opinion modeling, behavioral simulation, and longitudinal forecasting. Surprisingly, we discover strong compute scaling in all three settings, using a suite of 85 transformer LLMs with the Qwen3 architecture pre-trained on the DCLM web text corpus under fixed-compute budgets from $10^{18}$ to $10^{20}$ FLOPs. Then we evaluate 35 larger and more capable open-weight models up to 70B parameters, allowing us to predict downstream accuracy from loss. This reveals that the majority of behavioral and opinion simulation tasks will rapidly improve with scale, particularly when they involve populations that are well-represented in English web corpora. Longitudinal forecasting and underrepresented opinions scale more slowly, especially when they are less correlated with general knowledge and reasoning benchmarks like MMLU. In behavior simulation, scaling fails to improve model calibration with human cognitive biases like risk aversion, as well as human heuristics like learning correlated rewards from related tasks. On these tasks, even fine-tuned models fail to noticeably scale up performance from 0.5B to 8B parameters. Taken together, we conclude that scale will improve social simulations in most settings, but outliers exist, and improvements will be less reliable in low-resource domains.
Stanford HAI