Search papers, labs, and topics across Lattice.
This study evaluates how different expressions of belief (EoBs) influence large language models' (LLMs) adherence to user-provided context versus their prior knowledge. By developing a typology based on four linguistic dimensions鈥攆orm, evidentiality, epistemic stance, and tone鈥攖he authors create a benchmark to assess the response behavior of 16 LLMs across various architectures and scales. The findings indicate that larger and instruction-tuned models are generally less responsive to context, while certain EoBs are more effective in persuading LLMs, highlighting the importance of linguistic framing in model interactions.
Larger LLMs may ignore user context more than smaller models, revealing critical insights into how linguistic framing can sway model responses.
Users frequently express their beliefs to large language models (LLMs). In some situations, the LLM should accept these contextual beliefs as true. In others, they should stick to their prior knowledge. Notably, users'expressions of belief (EoBs) can take linguistically diverse forms - using presuppositions, evidential and certainty markers, or varied tones - each of which may have a different persuasiveness over the LLMs. We introduce a typology to systematically evaluate how different EoBs affect whether models follow context versus prior knowledge. The typology is grounded in four linguistically motivated dimensions: form, evidentiality, epistemic stance, and tone, spanning 17 fine-grained types. By pairing these EoBs with world knowledge facts, we generate controlled EoB-query pairs that isolate the effect of linguistic variation. Using this benchmark, we evaluate 16 LLMs that differ in architecture (Llama3, Qwen3, Gemma3), scale (1B-30B parameters), and training stages (base vs instruct). We identify meaningful variations in response behavior across these axes, e.g., that bigger models and instruction models tend to be less context-following than smaller models and base models. We further identify specific EoBs that statistically significantly persuade LMs more consistently than others. Our work reveals systematic patterns in how linguistic framing affects LLM context integration, with implications for prompt engineering and model robustness.
ETH