Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces a human-LLM collaborative annotation framework to refine speaker-attribute labels in multilingual text, addressing inherent ambiguity in demographic and social cues. The framework uses LLMs to identify recurring annotation rationales through expert interaction and employs disagreement-focused sampling for targeted re-annotation. Using this framework, the authors construct WhoSaidIt, a new multilingual dataset covering nine speaker-attribute labels, and benchmark LLM performance, revealing significant cross-lingual annotation differences and the impact of explicit rationales.
Human-LLM collaboration can dramatically improve the consistency of speaker-attribute annotation in multilingual contexts, revealing substantial cross-lingual differences previously obscured by noisy labels.
Annotating speaker attributes from text is inherently ambiguous, particularly in multilingual settings where demographic and social cues are implicit and culturally variable. We propose a human-large language model (LLM) collaborative re-annotation framework for stabilizing multilingual speaker-attribute labels under practical resource constraints. Starting from a noisy corpus, we use LLMs to surface recurring annotation rationales through iterative interaction with experts, and apply disagreement-focused sampling for targeted re-annotation. Using this framework, we construct WhoSaidIt, a multilingual dataset covering nine speaker-attribute labels. We quantify divergence between original and revised annotations, benchmark recent LLMs, and analyze the effect of explicit rationales on model behavior. Our results reveal substantial cross-lingual differences in annotation decisions and demonstrate both the strengths and limitations of LLMs in speaker-attribute classification.