Track the frontier, from condensed matter to cosmology.
Search papers, labs, and topics across Lattice.
This paper investigates whether Large Language Models (LLMs) can learn to control their internal activation states through in-context neurofeedback, a process where the model receives feedback on its activation scores and attempts to modify them. By disentangling the model's generation capabilities from its potential activation control, the study finds that while some models can significantly alter their activation scores, this effect is often attributable to a separate "writer" model's competence rather than genuine self-control. However, a subset of models demonstrates limited, albeit consistent, ability to modulate specific activation scores when presented with a fixed context, suggesting nascent forms of internal state manipulation.
LLMs can learn to manipulate their own internal activations, but this control is limited and often mimicked by external generation.
Probes on activation vectors are widely used to detect misbehavior in language models, but their success relies on the assumption that models cannot control what their activations encode. Humans can learn, through neurofeedback, to control the firing of even single neurons, and recent studies investigated whether this ability extends to language models, reaching conflicting conclusions. No experimental design so far disentangles competence at the task from genuine access to one's own activations. We fill this gap with in-context neurofeedback sessions, in which a "subject" model is asked to maximize (or minimize) a score derived from its activations, shown back to it at every turn. In our first experiment, the sentences are generated either by the subject or by a second "writer" model, while the score always derives from the activations of the subject. We test ten open-weight models from five families, with sizes from 9B to 32B. Some models move their own scores considerably, but no more than a writer of comparable competence moves them. Is this because the task requires no privileged access, or because the subject controls its activations while processing the sentences, whoever wrote them? To answer this, we prefill a fixed sentence as the subject's reply at every turn and ask it to increase or decrease the score. Three out of ten models show small but consistent control effects. We further ask whether models can control their activations to the point of fooling an activation monitor. We make models repeat sentences a monitor would flag, such as a deceptive answer for a deception monitor, and ask them to raise or lower the monitor's score. No model learns to lower it, but one model can raise it. In-context control of a model's own activations might not yet be a threat to activation monitors, but it exists in a few models and needs to be tracked as models evolve.