Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study examines how the inclusion of annotator demographic attributes in prompts affects the alignment between large language model (LLM) outputs and human annotations across five distinct tasks. The findings indicate that alignment peaks with one to three high-signal attributes, while using a full set of demographic attributes leads to decreased alignment, highlighting an over-specification threshold. Additionally, the research uncovers that the effectiveness of demographic prompting is context-dependent, influenced by the quality of attribute signals, task characteristics, and model architecture.
Overloading LLM prompts with demographic attributes can actually reduce alignment with human judgments, revealing a critical over-specification threshold.
We investigate how annotator demographic attributes, supplied as prompt cues, shape the alignment between large language model (LLM) predictions and human annotations across five tasks. Using five open-source LLMs, we systematically vary the number and composition of demographic components in the prompt, spanning every combination from single-attribute through full-attribute configurations. Our experiments reveal three principal findings. First, alignment consistently peaks with one to three high-signal attributes and degrades under the full attribute set, establishing a clear over-specification threshold. Second, the overall magnitude of demographic influence on human annotations does not predict which attributes improve LLM alignment; instead, both the learnability and the directional coherence of each attribute's annotation signal need to be considered jointly. Third, neuron probing reveals that specialized activation correlates with alignment gains only under coherent annotation signals, and that activation volume alone does not imply steerability. Together, these results demonstrate that demographic prompting is not a monolithic intervention: its utility is highly context-dependent, shaped by attribute signal quality, task characteristics, and model architecture.