Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper adapts stated-preference economics' validity framework to evaluate language models on questions lacking ground truth, employing concepts like content, construct, and criterion validity to assess response coherence. By applying these economic validity tests, derived from theoretical predictions such as downward-sloping demand, to a water-quality valuation survey administered to six LLMs, the study sharply differentiates model performance. The results demonstrate that while passing these validity tests indicates coherent responses, it does not guarantee factual correctness.
Stated-preference economics offers a robust framework for evaluating LLM responses to subjective questions, revealing that even advanced models can fail basic economic validity tests.
Many of the questions now put to large language models have no correct answer to score against: what a policy is worth, which option a user should choose, how to weigh competing values. Stated-preference economics has faced this problem for decades. It judges survey responses without knowing the true value, through a framework of validity and related concepts: content, construct, and criterion validity, reliability, incentive compatibility, and consequentiality. We argue that this framework is a general method for evaluating language models, and we set out what each concept means for LLM evaluation. We demonstrate the approach using a published water-quality stated preference economic valuation survey (Vossler et al. 2023) administered to six models. In this economic application, the validity tests take the form of predictions from economic theory: demand should slope down, and willingness to pay should respond to the scope of the good and to income. The tests separate the models sharply. Two older models fail the most basic test at a household income level of \$75,000, and the two newest pass every test of theoretical validity we can score, but diverge on convergent validity. Passing validity tests shows that a model's answers are coherent, not that they are correct.