Search papers, labs, and topics across Lattice.
This study evaluates the effectiveness of eight open-weight large language models (LLMs) in replicating resident attitudes towards affordable housing projects, specifically examining how support varies by distance from proposed developments and across different demographic groups. While Qwen 2.5 14B closely matched human responses in aggregate, it failed to accurately represent the underlying spatial and identity-conditioned structures, particularly in its handling of partisan differences. The findings highlight that LLMs can yield misleading results by approximating average effects while neglecting critical within-group heterogeneity and measurement stability, underscoring the need for more nuanced evaluations in urban planning contexts.
LLMs can replicate aggregate survey results but often miss the critical spatial and demographic nuances that shape public opinion on urban development.
There is growing interest in using large language models (LLMs) as low-cost proxies for resident attitudes in urban planning. Previous work shows that LLMs can predict average results of survey experiments, but less is known about whether they preserve the spatially anchored, identity-conditioned structure behind those averages, namely how support changes as a project approaches homes and how that response divides across tenure and partisan groups. We compared eight open-weight LLMs with 843 respondents in a US affordable-housing survey experiment, testing whether they reproduced the owner-renter difference in support change as a proposed development moved from 2 miles to 1/8 mile. Qwen 2.5 14B was closest (-0.242 versus the human -0.285) and was the only model to meet the prespecified +/-0.20 equivalence criterion; Phi-4 14B was directionally aligned but attenuated (-0.150), and other models showed weak, null, or reversed moderation. This aggregate match masked structural failure. Qwen attenuated the Republican contrast and exaggerated the Independent one, its RMSE across 27 party-by-tenure-by-item cells was 0.613, its median model-to-human variance ratio was 0.099, and question order shifted the contrast by +0.367. Identity-cue removal and selective nonresponse changed which comparisons were estimable, and rationale-first responses differed from matched direct-choice responses in 20.6-35.3% of focal comparisons. An LLM can thus approximate one aggregate contrast while failing to preserve the population structure, within-group heterogeneity, and measurement stability that generate it. Model evaluation in urban planning should test whether this spatial and social structure survives simulation, not only average effects.
