Search papers, labs, and topics across Lattice.
This paper introduces a distributional framework using Semantic Optimal Transport (SOT) to estimate semantic distances between sparse autoencoder (SAE) features across different layers, addressing challenges in feature matching and circuit compression. SOT represents each feature as an activation-weighted distribution over hidden states, projected into a shared reference space and compared using Wasserstein distance. The method demonstrates invariance to activation rescaling, stability under perturbations, and superior performance compared to decoder-vector and LLM-based baselines in capturing functional distinctions and compressing feature circuits.
Forget relying on decoder vectors or LLMs for SAE feature analysis – Semantic Optimal Transport offers a provably robust and empirically superior way to match features and compress circuits by modeling features as distributions over hidden states.
Sparse autoencoders (SAEs) have become a central tool for interpreting language models. However, two key SAE analyses that remain difficult to scale are (1) matching semantically similar features across multi-layers and (2) compressing large feature circuits into interpretable supernodes. Although these have been treated as separate problems, we show that both are instances of a more fundamental challenge, which we frame as the estimation of semantic distances between SAE features that lie on different activation manifolds. We introduce a distributional framework for this problem, in which each feature is represented not by a single decoder vector like in the literature, but by an activation-weighted distribution over the hidden states that express it. By projecting these distributions into a shared reference space and comparing them with Wasserstein distance, our method provides a unified semantic metric for cross-layer feature comparison. We prove that our representation is invariant to activation rescaling, stable under perturbations, and recovers true matches under finite-sample margin conditions. Empirically, our method outperforms decoder-vector and LLM-based baselines and captures subtle functional distinctions between related features. Notably, our method compresses large feature circuits into interpretable supernodes automatically.
