Search papers, labs, and topics across Lattice.
The paper introduces RADIUS, a two-dimensional alignment suite for evaluating LLM-based survey simulations by assessing both ranking and distribution alignment with human responses. It addresses the shortcomings of existing evaluation metrics that are often ad hoc, fragmented, and fail to capture ranking preferences. RADIUS incorporates statistical significance testing and provides an open-source implementation, enabling more robust and reproducible evaluation of survey simulations.
LLM-generated survey responses can be statistically accurate yet still miss the option most preferred by humans, highlighting a critical flaw in current evaluation methods.
Simulation of surveys using LLMs is emerging as a powerful application for generating human-like responses at scale. Prior work evaluates survey simulation using metrics borrowed from other domains, which are often ad hoc, fragmented, and non-standardized, leading to results that are difficult to compare. Moreover, existing metrics focus mainly on accuracy or distributional measures, overlooking the critical dimension of ranking alignment. In practice, a simulation can achieve high accuracy while still failing to capture the option most preferred by humans - a distinction that is critical in decision-making applications. We introduce RADIUS, a comprehensive two-dimensional alignment suite for survey simulation that captures: 1) RAnking alignment and 2) DIstribUtion alignment, each complemented by statistical Significance testing. RADIUS highlights the limitations of existing metrics, enables more meaningful evaluation of survey simulation, and provides an open-source implementation for reproducible and comparable assessment.
Amazon Science