Search papers, labs, and topics across Lattice.
This study explores the interaction between model-internalized priors and user-provided instructions in Large Language Models (LLMs) during zero-shot annotation tasks, specifically focusing on toxicity detection across various datasets. The authors find that nearly two-thirds of zero-shot errors are resistant to correction, with only 34.8% of initial errors amendable through additional prompting, and that high-confidence errors are particularly stubborn. Introducing the metric of Definition-Specific Familiarity (DSF), they demonstrate a significant positive correlation between this alignment measure and model performance, in contrast to traditional memorization metrics that do not predict success.
Two-thirds of zero-shot errors in LLMs resist correction, revealing the critical role of definition alignment over mere text memorization.
Large Language Models (LLMs) are increasingly used for zero-shot annotation and LLM-as-a-judge tasks, yet their reliability hinges on how model-internalized priors interact with user-provided instructions. We investigate three dimensions of this interaction: (1) how an LLM's familiarity with data and task definitions affects performance, (2) the extent to which additional information in prompts can correct zero-shot errors ("decision stickiness"), and (3) model susceptibility to misaligned task definitions. Through experiments on toxicity detection across diverse datasets (spanning social media, gaming, news, and forums) using both dense and mixture-of-experts models, we find that nearly two-thirds of zero-shot errors are resistant to correction, with an overall rescue rate (fraction of initial errors corrected by prompting) of only 34.8%. High-confidence errors prove especially resistant to correction. When given misaligned definitions, LLMs follow them while maintaining confidence levels unchanged from the aligned condition. Crucially, we introduce Definition-Specific Familiarity (DSF), which measures alignment between a model's internal concept and the task definition. After controlling for dataset-level confounds, DSF shows a positive association with model performance (partial r = +0.41), while three distinct memorization metrics (ROUGE-L, BERTScore, and embedding cosine similarity) all fail to show a positive association. These findings show the limitations of prompt-based correction in annotation tasks, highlighting the importance of definition alignment over text-level memorization.
