Search papers, labs, and topics across Lattice.
This paper introduces GAPS, a method for dimension-level conditioning in activation steering, which selectively intervenes on neurons carrying relevant concept information to suppress undesired behaviors in language models. By employing a static separability gate and a dynamic posterior gate, GAPS enhances the effectiveness of existing conditional methods while maintaining a low computational overhead. The results demonstrate that GAPS significantly reduces toxicity rates in language models, outperforming traditional token-level approaches and achieving a notable reduction from 6.52% to 0.48% in toxicity for Gemma-3.
GAPS slashes toxicity rates in language models by intelligently steering only the neurons that matter, achieving a remarkable reduction from 6.52% to 0.48%.
Activation steering suppresses undesired behaviors in language models by adding a steering vector to the hidden state during generation. Recent conditional methods such as CAST and DSAS improve the behavior-capability trade-off by deciding when to intervene, but once active, they apply the full dense vector to all hidden dimensions, regardless of whether a neuron carries concept information or already lies in the desired regime. We introduce dimension-level conditioning as a complementary axis of selectivity that also decides which neurons to intervene on. Our method, GAPS (Gated Activation steering via Posterior and Separability), combines two training-free gates: a static separability gate that restricts steering to neurons with statistically reliable concept information (via AUROC), and a dynamic posterior gate that steers a neuron only when its current activation is better explained by the undesired concept under a Gaussian model. The gates add O(D) overhead per token, and they plug into existing conditional methods. On toxicity mitigation (RealToxicityPrompts) and concept removal (OneSeC) with Gemma-3 (4B) and Qwen-3 (1.7B), GAPS consistently matches or improves the Pareto front of its token-level counterparts; under a fixed capability budget, DSAS+GAPS reduces Gemma-3's toxicity rate from 6.52% to 0.48%, versus 3.52% for DSAS alone. Ablations attribute most of the gain to the posterior gate.
