Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces a causal mediation analysis to pinpoint the factors within LLMs that lead to harmful content generation. The analysis reveals that harmful generation primarily originates in later layers, specifically within MLP blocks rather than attention mechanisms. Furthermore, the study identifies a sparse set of neurons in the final layer acting as a gating mechanism that determines the generation of harmful content based on signals propagated from earlier layers.
Harmful content generation in LLMs isn't a holistic property, but rather a localized phenomenon driven by specific neurons in later MLP layers acting as "harmfulness gates."
Large Language Models (LLMs) have been shown to generate harmful content. However, the underlying causes of such behavior remain under explored. We propose a causal mediation analysis-based approach to identify the causal factors responsible for harmful generation. Our method performs a multi-granular analysis across model layers, modules (MLP and attention blocks), and individual neurons. Extensive experiments on state-of-the-art LLMs indicate that harmful generation arises in the later layers of the model, results primarily from failures in MLP blocks rather than attention blocks, and is associated with neurons that act as a gating mechanism for harmful generation. The results indicate that the early layers in the model are used for a contextual understanding of harmfulness in a prompt, which is then propagated through the model, to generate harmfulness in the late layers, as well as a signal indicating harmfulness through MLP blocks. This is then further propagated to the last layer of the model, specifically to a sparse set of neurons, which receives the signal and determines the generation of harmful content accordingly.