Search papers, labs, and topics across Lattice.
This study investigates how user identity influences the moral evaluations of large language models (LLMs) through a behavioral approach, utilizing a structured conversational protocol across 12,000 interactions with two non-reasoning models. The findings reveal that moral judgments, particularly for contestable rule-governed acts, shift according to the user's professional role, indicating that user identity can condition LLM responses in significant ways. This research challenges traditional notions of static moral principles in AI value alignment, suggesting a need for dynamic moral frameworks that account for contextual influences.
User identity can dramatically shift LLM moral evaluations, revealing a troubling layer of contextual conditioning in AI responses.
This study adopts a behavioural bottom-up approach to AI value alignment to investigate whether an implicitly conveyed user identity shifts the moral evaluations of large language models (LLMs). Through a structured, multi-turn conversational protocol across 12,000 interactions, we evaluate AI value alignment in two non-reasoning models, gpt-4.1-mini-2025-04-14 and gemini-2.5-flash-lite. Rather than instructing the models to adopt a persona or prompting them with explicit moral stances, the user's professional role is introduced purely through value-neutral reasoning. The models are then asked for wrongness ratings from 0-100 on ten common-morality rules from Gert's moral framework. The results show that moral judgments vary with the user's role across both models. While grave-harm acts like killing exhibit a strong ceiling effect, contestable rule-governed acts demonstrate role-conditioned shifts that mirror the relationship between the user's profession and the act being rated. These findings demonstrate that unintended contextual conditioning via user identity permeates LLM moral evaluations, posing questions for the AI value alignment discourse regarding how to define acceptable bounds for role-based moral divergence. By doing so, the results contribute to reframing the AI value alignment discourse by suggesting future research on dynamic moral bounds rather than static moral principles or rules as frame of reference.
