Search papers, labs, and topics across Lattice.
This paper introduces VIBE, a novel benchmark designed for entity-centered affective profiling of large language model outputs, utilizing a Valence-Arousal-Dominance (VAD) framework. The study reveals that scalar favorability does not encompass the dimensions of arousal and dominance, and highlights the distinct nature of whole-response versus target-directed VAD, emphasizing the variability in affective tone based on context. The findings advocate for a standardized approach to reporting affective profiles, incorporating metadata to enhance interpretability and accountability in LLM outputs.
Scalar favorability fails to capture the full spectrum of affective nuances, revealing the need for a more sophisticated profiling approach in LLM evaluations.
Large language models routinely describe socially salient targets, including political figures, countries, religions, organizations, historical events, and social groups, encoding affective framing alongside factual content: a target may appear favorable or threatening, calm or conflictual, powerful or vulnerable. Existing work captures parts of this space through sentiment, favorability, and emotion benchmarks, but none combines target-directed VAD attribution, an explicit scorer contract, and a passport reporting format. We introduce VIBE, a benchmark for entity-centered affective profiling of LLM outputs in Valence-Arousal-Dominance (VAD) space. Its core contribution is a measurement contract: VIBE separates generation from external scoring, distinguishes scalar favorability, response-level VAD, and target-directed VAD, and reports profiles through an Affective Passport. Three empirical layers support the contract. H1 shows scalar favorability does not subsume arousal and dominance: valence findings are cross-validated (rV = 0.944 judge-human, rV = 0.954 inter-scorer); arousal and dominance are single-scorer directional estimates, not point-precise, consistent with known inter-annotator difficulty on these axes (rA = 0.495, rD = 0.702 among human annotators). H2 shows whole-response and target-directed VAD are different contracts: the same text can carry one affective tone overall while representing the named target differently. H3 is a protocol-drift diagnostic: elicitation conditions shift profiles, motivating context metadata in every affective report. These results motivate entity-centered affective profiling as a documented practice: profiles should be released with scorer identity, coverage, protocol, and interpretation limits.
