Search papers, labs, and topics across Lattice.
This paper introduces H-Res (Hierarchical Residual Steering), a novel mechanism for adapting large Transformer models as Dense Associative Memories without modifying synaptic weights or clogging retrieval buffers. By treating adaptation as a control problem on the activation manifold, H-Res effectively modulates the energy landscape to guide token trajectories into task-specific basins of attraction. The method not only preserves attention entropy but also outperforms traditional weight modification approaches by 26% on associative retrieval tasks while eliminating the overhead associated with prompt-based methods.
H-Res achieves a 26% improvement in associative retrieval tasks by steering token trajectories without altering the model's global equilibrium.
Large Transformer models function as Dense Associative Memories (DAMs), retrieving knowledge via high-dimensional attractor dynamics driven by the self-attention mechanism \citep{ramsauer2020hopfield, wu2024attention}. However, adapting these frozen memory systems to new tasks presents a fundamental ``Plasticity-Stability'' dilemma. Current methods either risk catastrophic interference by modifying synaptic weights directly (e.g., LoRA) \citep{hu2021lora} or degrade associative capacity by clogging the retrieval buffer with static prompt tokens (e.g., VPT) \citep{jia2022vpt}. In this work, we propose \textbf{H-Res} (Hierarchical Residual Steering), a mechanism that modulates the effective energy landscape of the Transformer without altering its global equilibrium or expanding its sequence length. By formulating adaptation as a control problem on the activation manifold \citep{chen2018neuralode}, H-Res learns a state-dependent vector field that steers token trajectories into task-specific basins of attraction. We formally prove that H-Res preserves the attention entropy of the foundation model and facilitates Neural Collapse \citep{papyan2020prevalence}. Empirically, Manifold Steering outperforms global weight modification by 26\% on associative retrieval tasks and eliminates the computational overhead of prompt-based methods, scaling effectively to structured domains \citep{zha2023vtab}.
