Search papers, labs, and topics across Lattice.
This paper introduces SAEVerbalizer, a novel framework that enhances the interpretability of sparse autoencoder (SAE) features by integrating SAE decoder directions into large language model (LLM) representations and fine-tuning the LLM to generate natural-language explanations. This approach addresses the inefficiencies of relying on external observations for feature explanation by enabling direct verbalization of features from the decoder directions. Experimental results demonstrate that the verbalization capability not only generalizes to unseen features but also effectively transfers across different SAE dictionaries and LLMs, showcasing the framework's robustness and versatility.
Directly verbalizing sparse autoencoder features from LLM representations transforms how we interpret model behavior, making explanations more efficient and insightful.
Sparse autoencoders (SAEs) are proposed to extract numerous features from large language model (LLM) representations, yet explaining these features still relies primarily on external observation. This reliance leads to superficial explanations inferred from observed model behavior and computational inefficiency from collecting such behavioral evidence at scale. We introduce SAEVerbalizer, a framework that injects SAE decoder directions into an LLM's representations and fine-tunes the LLM's downstream layers to generate natural-language explanations of the injected features. Once trained, the resulting verbalizer explains SAE features directly from decoder directions, addressing both limitations. Our experiments show that the learned verbalization capability generalizes to unseen features, transfers across separately trained SAE dictionaries, and, with a lightweight adapter, extends to SAE features from different LLMs. Intervention experiments show that injecting multiple directions yields an explanation combining their meanings, while reversing individual directions produces corresponding meaning shifts.
Tsinghua AI