Search papers, labs, and topics across Lattice.
This paper introduces GEMS, a training-free method that addresses the collapse of model behavior when multiple semantic directions are superposed by applying geometric constraints to activation steering. The authors identify two primary sources of collapse鈥攄istributional deviation and directional interference鈥攁nd propose specific geometric interventions to mitigate these issues. Experimental results demonstrate that GEMS maintains high accuracy on GSM8K with concurrent non-mathematical direction injections, achieving 98% accuracy compared to a baseline of 92%, while also showing minimal performance degradation on Wikitext-2.
GEMS enables LLMs to maintain high accuracy even when injecting multiple semantic directions, defying the collapse typically observed in such scenarios.
Activation steering controls model behavior by modifying intermediate hidden states at inference time without retraining. Existing methods handle only single-direction injection; when multiple semantic directions are superposed without constraints, the model collapses. We show that this collapse decomposes into two independently acting sources: distributional deviation, where additive perturbations accumulate in norm across layers and drive activations outside the training distribution, and directional interference, where non-orthogonal semantic vectors mutually dampen when superposed. These two sources define the design constraints that any training-free multi-directional intervention must address. As one instantiation of these principles, we propose GEMS, a training-free method that maps each source to a corresponding geometric constraint: norm-preserving weighted superposition and targeted attention-pathway injection for distributional deviation, and real-time orthogonalization for directional interference. On GSM8K, injecting three concurrent non-mathematical directions preserves accuracy at 98% (baseline 92%), while unconstrained addition collapses to 4%; on Wikitext-2, the same injection incurs only 2.2% PPL increase. Component ablation isolates the causal role of each constraint, and layer-level probes confirm that orthogonalized signals survive the FFN pathway and reach the output distribution with semantic specificity. Qualitative steering effects transfer across architectures from 3B to 31B.
