Search papers, labs, and topics across Lattice.
The paper introduces a unified theoretical framework for activation steering in LLMs based on ordinary differential equations (ODEs), addressing limitations of existing methods by providing a principled approach for designing steering directions. They demonstrate that conventional activation addition is a first-order approximation of an ODE solution and propose ODESteer, an ODE-based steering method guided by barrier functions derived from control theory. ODESteer, which identifies steering directions using the log-density ratio between positive and negative activations, achieves significant empirical improvements on LLM alignment benchmarks, including a 5.7% improvement on TruthfulQA.
LLMs can be steered more effectively by viewing activation manipulation through the lens of ordinary differential equations and control theory, leading to significant gains in alignment benchmarks.
Activation steering, or representation engineering, offers a lightweight approach to align large language models (LLMs) by manipulating their internal activations at inference time. However, current methods suffer from two key limitations: \textit{(i)} the lack of a unified theoretical framework for guiding the design of steering directions, and \textit{(ii)} an over-reliance on \textit{one-step steering} that fail to capture complex patterns of activation distributions. In this work, we propose a unified ordinary differential equations (ODEs)-based \textit{theoretical} framework for activation steering in LLM alignment. We show that conventional activation addition can be interpreted as a first-order approximation to the solution of an ODE. Based on this ODE perspective, identifying a steering direction becomes equivalent to designing a \textit{barrier function} from control theory. Derived from this framework, we introduce ODESteer, a kind of ODE-based steering guided by barrier functions, which shows \textit{empirical} advancement in LLM alignment. ODESteer identifies steering directions by defining the barrier function as the log-density ratio between positive and negative activations, and employs it to construct an ODE for \textit{multi-step and adaptive} steering. Compared to state-of-the-art activation steering methods, ODESteer achieves consistent empirical improvements on diverse LLM alignment benchmarks, a notable $5.7\%$ improvement over TruthfulQA, $2.5\%$ over UltraFeedback, and $2.4\%$ over RealToxicityPrompts. Our work establishes a principled new view of activation steering in LLM alignment by unifying its theoretical foundations via ODEs, and validating it empirically through the proposed ODESteer method.
AI2
Amazon Science
UW