Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces the Tversky Monosemanticity Score (TMS), a novel label-free metric designed to assess the monosemanticity of Sparse Autoencoders (SAEs) by measuring the coherence of activation sets from binarized latents. The significance of TMS lies in its ability to evaluate explanation quality without relying on external concept labels or pretrained embedding models, thus reducing sensitivity to encoder geometry. Experimental results demonstrate that TMS outperforms existing metrics in terms of robustness against encoder anisotropy while maintaining alignment with established indicators of monosemanticity, revealing unique training dynamics across different SAE configurations.
TMS reveals that traditional metrics for assessing monosemanticity in Sparse Autoencoders are vulnerable to encoder geometry, highlighting a more robust, label-free alternative.
Within Explainable Artificial Intelligence, mechanistic interpretability uses Sparse Autoencoders (SAEs) to extract more interpretable features from neural representations. However, assessing their monosemanticity, and thus explanation quality, remains challenging. Existing metrics require external concept labels or depend on pretrained embedding models, making them sensitive to encoder's geometry. We introduce the Tversky Monosemanticity Score (TMS), a label-free metric that operationalizes monosemanticity as activation-set coherence of binarized SAE latents, and does not require external embedding encoders. We evaluate TMS on SAEs trained on features from pretrained vision and vision-language models (DINOv3, CLIP, BLIP2), two common SAE regimes (TopK, BatchTopK), multiple sparsity levels, and expansion factors. Our results show that TMS is less affected by encoder anisotropy than its embedding-based alternative, while remaining aligned with established monosemanticity indicators. TMS also reveals distinct SAE training dynamics across base models. Moreover, under encoder anisotropy, TMS provides a stronger indication of probe-based concept deletion effectiveness, while being competitive otherwise.