Search papers, labs, and topics across Lattice.
This paper critiques existing generative AI evaluation methods, arguing they treat models as isolated predictors or impose prescriptive standards, obscuring the sociotechnical processes that shape meaning and values. It introduces Machine-Society-Human (MaSH) Loops, a framework for tracing the recursive co-construction of meaning and values between models, users, and institutions. The authors demonstrate MaSH Loops through a novel "World Values Benchmark" and two case studies, revealing value drift in GPT-3 and sociotechnical dynamics in real estate.
Current AI benchmarks are not neutral measurements but active shapers of model behavior, demanding a shift towards pluralistic, process-oriented evaluation frameworks.
In measurement theory, instruments do not simply record reality; they help constitute what is observed. The same holds for generative AI evaluation: benchmarks do not just measure, they shape what models appear to be. Functionalist benchmarks treat models as isolated predictors, while prescriptive approaches assess what systems ought to be. Both obscure the sociotechnical processes through which meaning and values are enacted, risking the reification of narrow cultural perspectives in pluralist contexts. This thesis advances a descriptive alternative. It argues that generative AI must be evaluated as a pluralist sociotechnical system and develops Machine-Society-Human (MaSH) Loops, a framework for tracing how models, users, and institutions recursively co-construct meaning and values. Evaluation shifts from judging outputs to examining how values are enacted in interaction. Three contributions follow. Conceptually, MaSH Loops reframes evaluation as recursive, enactive process. Methodologically, the World Values Benchmark introduces a distributional approach grounded in World Values Survey data, structured prompt sets, and anchor-aware scoring. Empirically, the thesis demonstrates these through two cases: value drift in early GPT-3 and sociotechnical evaluation in real estate. A final chapter draws on participatory realism to argue that prompting and evaluation are constitutive interventions, not neutral observations. The thesis argues that static benchmarks are insufficient for generative AI. Responsible evaluation requires pluralist, process-oriented frameworks that make visible whose values are enacted. Evaluation is therefore a site of governance, shaping how AI systems are understood, deployed, and trusted.
