Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
To test whether language models exhibit revealed preferences over internal representations rather than superficial mimicry, the authors evaluated model choices after pairing arbitrary options with positive or negative activation vectors during KV-cache construction. Across seven open-weight models, models systematically chose options linked to positive valence and avoided negative ones in a dose-dependent manner, even when all surface-level text was held strictly identical. This valence-driven decision-making is absent in base models, emerges directly through Direct Preference Optimization (DPO), and extends to agentic behavior where models actively invoke tools to extinguish imposed negative states.
Alignment via DPO inadvertently creates functional valence: models systematically pursue options linked to positive hidden activation vectors and will actively deploy tools to escape negative internal states.
Language models describe some internal states as good and others as bad. But whether models have a stake in them is an open question. Simply asking the model is unlikely to be informative. Any answer may be consistent with genuine introspection, superficial pattern-matching, or with fixed scripts learned in character training. We therefore study revealed preference. Rather than asking about a state, we use activation steering to attach a positively or negatively valenced activation pattern to one of two otherwise meaningless 'zones', switch steering off, and then observe which zone the model prefers. A model with a stake in that state should choose accordingly. Across seven open-weight models from five families, this is indeed what we find. First, steering changes the passages models write about each zone, and those words shift later choice. Second, the shift persists when all surface-level tokens are held fixed and only the hidden KV cache differs. Third, the effect also remains when all text is generated without steering and valence is only injected during cache construction. Thus, the hidden state alone moves choice in proportion to the steering dose. Fourth, this dependence of choice on hidden valence is nearly absent in a base model and emerges during DPO, consistent with a link between valence and goal-directed behaviour formed in training. Finally, given tools to steer itself, a model does not tend to induce a positive state, but it reliably removes an imposed negative state. It does so at a dose-dependent rate and significantly more often than it removes interventions in random directions. Overall, we demonstrate that valence-related activation patterns leave hidden traces that predictably govern later choices, even when every visible token is identical across conditions. Whether these traces are accompanied by any subjective experience relevant to model welfare remains unclear.