Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces the Multi-Agent Perspectivist Preference Optimization (MAP-PO) framework to address the inherent disagreements in sexism labeling by clustering annotators based on their labeling behavior. By fine-tuning individual Large Language Model agents for each cluster and optimizing preferences through a combination of individual and team-level rewards, the framework effectively captures diverse perspectives on sexism. The results demonstrate that cluster-specific training is essential for accurate representation, as agents trained solely on their cluster labels can exceed expected performance boundaries when guided by a shared training signal.
Clustering annotators by behavior reveals that tailored training can significantly enhance the accuracy of sexism detection models, surpassing traditional majority voting methods.
When people label text for sexism, they often disagree, and not because some of them are wrong: they genuinely perceive sexism differently. Most NLP systems discard this disagreement by collapsing it into a majority vote. We propose the Multi-Agent Perspectivist Preference Optimization (MAP-PO) framework to keep these different perspectives. On the EXIST 2024 dataset of labeled English and Spanish tweets, we first cluster annotators by their labeling behavior rather than their demographic attributes. We then fine-tune one Large Language Model agent per cluster to reproduce that cluster's annotation behavior, and coordinate the agents with preference optimization that combines individual and team-level rewards. We evaluate MAP-PO in four settings defined by two languages and two backbone language models, asking whether each agent reproduces the annotations of its own cluster and whether the agents together reproduce the majority label. Two findings hold in all four settings. First, without fine-tuning the agents behave almost identically, so cluster-specific training is necessary. Second, we show that training each agent only on the labels of its own cluster pushes the agents far beyond the clusters they should represent, while adding a shared team-level training signal consistently keeps each agent calibrated to its cluster.