Search papers, labs, and topics across Lattice.
This study investigates the ability of large language models (LLMs) to capture human judgment by addressing two key limitations: the failure to represent full response distributions and instability across different phrasings. By employing targeted prompting strategies, such as eliciting standard deviations and ensuring clarity in scenarios, the authors demonstrate significant improvements in AI-human alignment across diverse moral scenarios and beliefs. The findings reveal that while LLMs struggle with self-calibration of error estimates, they effectively track human variability, indicating that the quality of prompts can substantially enhance model performance.
Simple prompting techniques can transform LLMs into more reliable mirrors of human judgment, recovering the full spectrum of responses.
Are large language models (LLMs) bad at capturing human judgment? Two commonly stated limitations are that LLMs fail to capture full distributions of responses, and that their judgments are unstable across wording variations. We demonstrate simple prompting strategies that mitigate these limitations. Across two datasets--a U.S.-representative set of 144 moral scenarios and 38 moral beliefs from the International Social Survey Programme's Family and Changing Gender Roles module covering 32 countries--we show how simple elicitation techniques help improve AI-human alignment. First, prompting models to report standard deviations and response proportions recovers the full range of human responses better than common strategies. Second, ensuring scenarios are clear to human participants--as reflected in human confusion ratings--boosts model alignment, and LLMs can track human confusion ratings. At the same time, we find that LLMs' estimates of their own error are poorly calibrated, though they can predict human variability relatively well. These results suggest that asking better questions to LLMs can yield better answers.
Microsoft Research