Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces a domain-agnostic multi-layered framework for the unsupervised extraction of perspectives in text, aimed at evaluating the pluralistic gap in LLM-generated content. By applying this framework to a dataset of book reviews, the authors reveal that while certain models and prompting strategies can capture a wide range of viewpoints, they still fail to adequately represent rarer perspectives, leading to significant discrepancies with human-generated text. The findings underscore the challenges in achieving true pluralism in LLM outputs, highlighting the need for improved alignment strategies in AI-generated content.
Despite advances in LLMs, a significant portion of rare perspectives remains underrepresented, revealing a critical gap in pluralistic alignment.
The growing need to represent diverse perspectives has increased interest in pluralistic LLM generation. Although difficult to operationalize, identifying perspectives expressed in text would provide clear guidance on pluralistic alignment and more clearly articulate the pluralistic gap in LLM generation. While models have been shown to reduce the diversity of training data and generate homogeneously, this has been demonstrated primarily on multiple-choice questionnaires or using high-level characteristics of free-form text. In this paper, we introduce and implement a domain-agnostic multi-layered framework for unsupervised extraction of perspectives suitable for identifying the pluralistic gap in LLM-generated text. We evaluate our framework on book reviews, a highly opinionated dataset representing diverse perspectives, and compare various prompts and models. Our results show that while some models and prompting techniques come close to covering a broad spectrum of perspectives, rarer perspectives remain disproportionately underrepresented, resulting in distributions that diverge from human text.