Search papers, labs, and topics across Lattice.
This paper uses mechanistic interpretability via Circuit Tracer on Gemma-2-2b to understand how LLMs detect software vulnerabilities in C/C++ code. The analysis surprisingly reveals that LLMs primarily rely on "safety detectors" that recognize safe coding patterns, rather than directly detecting vulnerability signatures. Ablation experiments confirm the causal role of specific attention heads in early layers (L5, L7) and MLP neurons in Layer 7, demonstrating that vulnerability detection relies on sparse, interpretable circuits.
LLMs flag code as vulnerable not by spotting the danger, but by failing to recognize safety.
Large language models (LLMs) can detect software vulnerabilities, but how do they actually identify vulnerable code? We address this question using mechanistic interpretability; analyzing the internal computations of a neural network to understand its reasoning process.Using Circuit Tracer on Gemma-2-2b, we trace the computational pathways activated when the model classifies 472 C/C++ code samples as vulnerable or safe. Our analysis reveals a surprising finding: the model primarily relies on safety detectors, attention heads that recognize safe coding patterns, rather than directly detecting vulnerability signatures. When these safety detectors fail to activate, the model classifies code as vulnerable. We identify the critical neural components: specific attention heads in early layers (L5, L7) that focus on safety patterns, and Multilayer Perceptron (MLP) neurons in Layer 7 that encode vulnerability-related features. Ablation experiments confirm their causal role; removing Layer 11 drops vulnerability detection accuracy from 100% to 6%, while ablating just 20 neurons in Layer 7 reduces it by 50%.Our findings show that LLM vulnerability detection uses sparse, interpretable circuits (only 16% of model capacity), enabling circuit-level explanations for security predictions and targeted improvements to detection systems.
