Search papers, labs, and topics across Lattice.
This paper introduces the Structured Sparse AutoEncoder ($S^2AE$), which enhances the learning of modality-consistent concepts in vision-language models by enforcing semantic and spatial consistency among image patches. By implementing a structured sparsity regularization that promotes inter-group disentanglement and intra-group consistency, $S^2AE$ significantly improves semantic alignment and representational efficiency while preserving high reconstruction fidelity. The method demonstrates notable gains in neuronal monosemanticity across modalities, leading to more coherent and disentangled representations in multimodal feature learning.
Achieving a 6.06% boost in semantic alignment while maintaining over 99% reconstruction fidelity, $S^2AE$ redefines how we approach concept consistency in vision-language models.
Sparse autoencoders (SAEs) have emerged as a promising technique for mechanistic interpretability by learning a set of sparse latent features in large models, each of which encodes a distinct concept. However, in vision-language models (VLMs), vanilla SAEs struggle to learn modality-consistent concepts, with concepts often exhibiting fragmented coverage (i.e., disjoint regions) in the visual modality. To address this challenge, we propose a Structured Sparse AutoEncoder ($S^2AE$) that enforces concept consistency from both semantic and spatial perspectives in the visual modality. Specifically, we group image patches based on Transformer attention similarity and spatial proximity, and introduce a structured sparsity regularization when training the vanilla SAE. The regularization consists of exclusive sparsity for inter-group concept disentanglement and group sparsity for intra-group concept consistency, which drives the latent neurons by SAEs to specialize in distinct, semantically grounded concepts. Evaluated on the \texttt{Qwen2.5-VL-7B-Instruct} model, the method achieves 6.06% average improvement in semantic alignment (mIoU) and 60.81 in representational efficiency (lower l0 norm) while maintaining near-perfect reconstruction fidelity with an Explained Variance above 99%. Cross-modal analysis further demonstrates that $S^2AE$ enhances neuronal monosemanticity by this visual structural prior, achieving a 3.08% average gain in semantic consistency and a 2.37% average gain in monosemanticity scores for both modalities of multimodal features, thereby fostering more coherent and disentangled representations.
