Search papers, labs, and topics across Lattice.
To understand how stereotypical biases transition from internal representations to generated text, this work evaluates Llama-3.1-8B, Qwen3-8B, and Gemma-2-9B using linear probing, attribution patching, and sparse autoencoder (SAE) feature ablations across multiple languages. The authors uncover a massive mechanistic disconnect: probe decodability peaks 36–53% of model depth earlier than causal attribution patching, while only 6–18% of stereotypical residual-stream SAE features exhibit language-agnostic causal effects (and 0% are category-agnostic). These findings show that representational decodability is an unreliable proxy for downstream behavioral impact and that multilingual bias operates through fragmented, language-specific circuits rather than unified conceptual representations.
Where an LLM represents a stereotype is detached from where it acts on it: linear decodability peaks up to 53% of model depth earlier than causal attribution, while fewer than 18% of stereotype-associated SAE features transfer across languages.
Multilingual LLMs show stereotype-related behavior that varies across languages, but behavioral scores do not show where the relevant information is represented or how it affects the output. To investigate these internal mechanisms, we compare linear probing, attribution patching, sparse autoencoders (SAEs) and feature ablation in Llama-3.1-8B, Qwen3-8B, and Gemma-2-9B. Probe performance peaks substantially earlier than attribution in all three models, with a separation of 36-53% of model depth. Retained Llama-Scope features often match the social category on which they were selected and form recurring semantic families, but their lexical alignment and ablation effects vary across SAE suites. Only 6-18% of evaluated residual-stream features have language-agnostic effects under our criterion, and none are category-agnostic. Language-agnostic features have larger mean ablation effects in Llama-Scope, but this pattern does not repeat in the other SAE suites. Decodability, output influence, and cross-lingual ablation effects therefore need to be measured separately.
