Search papers, labs, and topics across Lattice.
This study investigates the phenomenon of visual lock-in in vision-language models, where outdated verbal descriptions can misguide visual judgments in changing scenes. The authors identify "Prior Directions," which are recurrent axes in model representations that lead to stronger lock-in effects when the model's representation changes in a compact manner. Their findings reveal that interventions targeting these Prior Directions can effectively restore accurate visual grounding, highlighting the structured nature of how prior information influences model behavior.
Visual lock-in can be reversed by targeting specific recurrent axes in model representations, revealing a structured influence of prior information on decision-making.
Vision-language models often use descriptions of earlier visual states to make decisions about the current scene. When the scene changes, stale language can redirect an otherwise correct visual judgment toward an outdated answer. We study this failure as visual lock-in in a controlled grounding setting where only the verbalized prior varies. Across models, stronger lock-in accompanies smaller changes in the model representation before the final answer. This reversal suggests that lock-in depends not on how far this representation moves, but on how that movement is organized. In models that are harder to correct, prior-induced changes concentrate along a compact set of directions that repeatedly appear across examples. We call these recurrent axes the Prior Directions. They recur on held-out examples, while a descriptive four-model comparison associates greater concentration with stronger lock-in. Controlled interventions show that removing the component aligned with the Prior Directions restores visual grounding, whereas removing an equally large orthogonal component has little effect. Prior control thus arises when prior-induced changes form a coherent and reusable pattern in the representation used to produce the answer. This account explains why the same prior remains revisable in one model yet becomes dominant in another.
