Search papers, labs, and topics across Lattice.
This study investigates the impact of persona conditioning on the sensitivity of large language models (LLMs) used as relevance assessors in information retrieval (IR) evaluation. By employing five distinct assessor roles derived from task-oriented personas, the research reveals that LLMs exhibit structured sensitivity to framing, with varying effects on assessment strictness and interpretation without causing significant relevance reversals. Notably, while high-capacity models maintain system-ranking agreement, smaller models demonstrate increased instability, highlighting the importance of persona roles over sources in evaluating LLM performance in IR contexts.
LLMs can be stress-tested for sensitivity to assessor framing, revealing that smaller models are particularly prone to instability in relevance judgments.
Large language models (LLMs) are increasingly used as relevance assessors in information retrieval (IR) evaluation, raising questions about how assessor framing affects judgment reliability and downstream system comparison. We study persona conditioning as a diagnostic mechanism for exposing LLM assessor sensitivity. Using task-oriented personas drawn from two complementary sources (PersonaHub and NVIDIA Nemotron-Personas-USA), we instantiate five assessor roles emphasizing intent interpretation, domain expertise, contrastive judgment, evidence verification, and global search-quality assessment, compared with a standard UMBRELA baseline. Across six LLM backbones on TREC DL20 and RAG24, our analyses reveal structured rather than uniform assessor sensitivity. Judgments usually remain close to the baseline while shifting assessment strictness, evidential threshold, or interpretation emphasis rather than producing widespread relevance reversals. At the system level, high-capacity models preserve system-ranking agreement, while smaller models amplify persona-induced instability. Local rank-displacement analysis shows sensitivity concentrates on particular retrieval systems and system types, especially neural ranking/reranking systems on DL20 and RAG-oriented pipelines on RAG24. Persona source matters less than assessor role and model capacity. These findings position persona-conditioned judging as a controlled sensitivity probe for stress-testing LLM-based IR evaluation pipelines and identifying systems whose evaluation outcomes are sensitive to assessor framing.
