Search papers, labs, and topics across Lattice.
This study introduces Distributed Sparse Interventions (DSI), a novel approach for investigating the causal influence of individual neurons on language model outputs by focusing on neuron-level nonlinearities and interactions. By activating task behavior through interventions on as little as 0.01% of neurons, DSI reveals substantial neuron-specific effects that traditional global steering methods overlook. The findings underscore the potential for fine-grained control over model behavior and enhance our understanding of how task-relevant computations are localized within neural networks.
Sparse interventions can activate complex task behaviors in language models by targeting just 0.01% of neurons, revealing hidden nonlinearities in model dynamics.
Language models perform a wide range of tasks at varying levels of abstraction with the capacity to flexibly infer tasks from context, execute multiple tasks simultaneously, and select among competing tasks. To study the role of model components in task behaviour, their causal influence can be investigated through interventions. Prior work on model steering has largely focused on interventions along global directions in activation space, modeling task representations as approximately linear and additive. By studying interventions at the neuron level, we find substantial, neuron-specific nonlinear effects on model outputs that are not captured by current steering approaches. We introduce Distributed Sparse Interventions (DSI), an intervention approach that considers nonlinearities and interactions between neurons across layers to identify sparse sets of neurons that elicit task-relevant computations. Across a range of tasks, we demonstrate that DSI can activate task behaviour in instruction-tuned language models by localising and intervening on as few as 0.01% of neurons, highlighting the effectiveness of sparse, distributed interventions in the neuron basis. Additionally, adopting a set-based perspective enables computations over the identified neuron sets, offering insights into the roles of individual neurons by analysing their effects across tasks. Through sparse interventions, DSI enables fine-grained control over model behaviour, localisation of task-relevant neuron sets, and furthers our understanding of task composition.
