Search papers, labs, and topics across Lattice.
This paper identifies and formalizes seven desiderata for evaluating NLP models designed to handle subjective judgments, focusing on user-centric impact and the nuanced representation of subjectivity. Through a review of 60 papers, the authors reveal that current evaluation practices often overlook key aspects of subjectivity, such as distinguishing between ambiguous and polyphonic inputs and ensuring effective expression of subjectivity to the user. The analysis highlights significant gaps in how subjectivity is currently assessed in NLP research.
Current NLP evaluations miss crucial aspects of subjectivity, potentially leading to models that fail to represent diverse perspectives effectively.
Subjective judgments are part of several NLP datasets and recent work is increasingly prioritizing models whose outputs reflect this diversity of perspectives. Such responses allow us to shed light on minority voices, which are frequently marginalized or obscured by dominant perspectives. It remains a question whether our evaluation practices align with these models'objectives. This position paper proposes seven evaluation desiderata for subjectivity-sensitive models, rooted in how subjectivity is represented in NLP data and models. The desiderata are constructed in a top-down approach, keeping in mind the user-centric impact of such models. We scan the experimental setup of 60 papers and show that various aspects of subjectivity are still understudied: the distinction between ambiguous and polyphonic input, whether subjectivity is effectively expressed to the user, and a lack of interplay between different desiderata, amongst other gaps.
