Search papers, labs, and topics across Lattice.
This study evaluates the ability of large language models (LLMs) to replicate human survey responses by focusing on distributional alignment rather than just mean-level agreement. Using a non-public consumer choice experiment on Korean instant noodle purchases, the authors find that while LLMs can reasonably replicate condition-level patterns, they significantly struggle to capture the underlying distributional structure of human responses. Notably, the models fail to outperform a simple baseline that matches the pooled human distribution, highlighting the limitations of mean-based evaluation metrics in assessing LLM performance.
LLMs may mimic human responses at the surface level but often miss the deeper distributional nuances, revealing a critical flaw in conventional evaluation methods.
LLMs are increasingly used to simulate human survey responses, but prior work has mainly evaluated replication using mean-level or aggregate agreement, offering limited insight into whether LLMs reproduce the variability of human behavior. We evaluate LLM-based survey replication at the distributional level using a non-public 2010 consumer choice experiment on Korean instant noodle purchases, a setting unlikely to overlap with model training data. We evaluate three response variables of differing statistical type: binary purchase incidence, categorical brand choice, and count purchase quantity. For each, we compare human and LLM responses at mean-level, pattern, and distributional alignment, and against reference baselines from the human data alone. LLMs reproduce condition-level patterns reasonably well but fail to capture distributional structure: for purchase quantity, no model beats a condition-insensitive baseline that simply matches the pooled human distribution. Because models that match human means well can still produce distributions further from humans than this baseline, mean-based evaluation alone can be actively misleading. Replication also varies with input configuration, with structured personas and multimodal inputs improving alignment while explicit reasoning prompting degrades it monotonically.
