Search papers, labs, and topics across Lattice.
This study investigates the effects of contextual information on stance detection in social media, focusing on how large language models (LLMs) perform with various types of contextual prompts. By evaluating multiple LLMs across four benchmark datasets, including a novel German Twitter dataset, the authors find that while LLM-generated target descriptions enhance accuracy, other contextual features can hinder performance due to noise. The research underscores the complexity of leveraging contextual information in zero-shot prompting, revealing that LLMs often misidentify relevant from irrelevant context, which can lead to degraded outcomes.
Contextual prompts can significantly boost stance detection accuracy, but only if the right type of context is chosen鈥擫LM-generated descriptions shine while user metadata may backfire.
Stance detection on social media is challenging due to short, noisy, and context-dependent language. While large language models (LLMs) show zero-shot generalization, they are typically prompted without contextual information, which limits their ability to interpret ambiguous posts. In this work, we systematically investigate the impact of incorporating real-world (e.g., user biographies), derived (e.g., political party), and LLM-generated (e.g., target descriptions) contextual features into zero-shot prompting for stance detection on Twitter. Our evaluation spans four benchmark datasets, including a new high-quality German Twitter stance dataset. Across multiple LLMs, we find that integrating contextual information improves performance, but only under specific conditions. LLM-generated target descriptions consistently enhance accuracy, while other user metadata has mixed or even detrimental effects. Notably, we show that the inclusion of other tweets by the same user, often beneficial in supervised learning, can impair performance due to input noise. Our qualitative analysis reveals that LLMs struggle to distinguish task-specific useful information from irrelevant context. Our findings highlight both the promise and challenges of prompting with context information in noisy real-world settings. We publish code and data at this \href{https://github.com/tilmanbeck/stance-context-twitter}{page}.
