Search papers, labs, and topics across Lattice.
This paper investigates how reinforcement learning (RL) shapes internal representations in language models by training models in a semantically neutral maze environment and extracting concept vectors for rewarded and punished trajectories. The key finding is that RL recruits a pre-existing "functional welfare axis," where the punishment vector aligns with negative emotions and promotes failure, while the reward vector acts as its mirror image. This welfare axis is shown to pre-exist RL training and can be activated even in pre-trained models, suggesting that RL leverages existing representations rather than creating new ones.
RL fine-tuning might be less about teaching LLMs new tricks and more about activating pre-existing "good vs. bad" representations lurking within them.
How does reinforcement learning shape a language model's internal representations? We present evidence that RL recruits a pre-existing representation of functional welfare: an estimate of how well or badly the system is doing, relative to its goals. We train several language models in a novel, semantically neutral maze environment. We then extract concept vectors for rewarded and punished trajectories, and evaluate those vectors in settings unrelated to the maze environment. The punishment vector behaves like a representation of negative welfare: it promotes failure and impossibility tokens, it aligns with negative emotion concepts, it negatively tracks goal-achievement, and steering with it induces negative self-reports, pathological backtracking, refusal, and uncertainty. The positive reward vector behaves as the mirror image, and the two are nearly antiparallel. These effects are robust when controlling for tile-to-reward mapping, scale, instruct tuning, RL training algorithm, model family, and LoRA versus full-finetuning, and largely persist when we replace RL with supervised fine-tuning. Importantly, the vectors are effective in models before they have undergone maze training. Combined with observations that the effects also appear in pretrain-only models, we therefore argue that this functional welfare axis pre-exists post-training: it is recruited, rather than created, by post-training. While we make no claims about any experience of welfare, the axis offers a demonstration that minimal reward signals can broadly affect model behavior by recruiting pre-existing welfare-like representations, with implications for interpretability, post-training dynamics, and alignment.
