Search papers, labs, and topics across Lattice.
This study explores how language models, specifically Qwen3-8B, internally assess the value of their current trajectory in achieving goals by constructing a "value" axis using synthetic in-context reinforcement learning data. The findings reveal that activations along this axis can effectively differentiate between varying levels of confidence, the necessity for backtracking, and the correctness of outputs. Notably, steering the model towards high value suppresses self-correction, while low value prompts exploration, indicating that models encode an estimate of expected goal success that influences their confidence in decision-making.
Language models can internally track their trajectory's value, influencing their confidence and decision-making in real-time.
We investigate whether language models internally track the value of their current trajectory, defined as the likelihood that their ongoing strategy will achieve their goals. Using synthetic, in-context reinforcement learning data, we construct a "value" axis for Qwen3-8B. We find that activations along this axis distinguish between high vs. low verbalized confidence, rollouts without and with backtracking, and correct vs. corrupted code. Steering towards high value causally suppresses self-correction and reduces explanatory verbosity, while steering towards low value induces backtracking and exploration. We demonstrate that direct preference optimization (DPO) can increase the internal value of rewarded behaviors (e.g. use a certain word), causing the model to act more confidently after exhibiting them. Finally, we apply the value axis to study in-the-wild settings. For example, we find that Qwen assigns low value to politically sensitive chat queries after post-training and that supervised fine-tuning increases internal confidence within the training domain. Our results suggest that language models linearly encode an estimate of expected goal success that modulates their confidence in pursuing a direction.
Stanford HAI