Search papers, labs, and topics across Lattice.
This paper introduces the Joint Sparse Autoencoder (JSAE), which enhances the controllability of cross-modal steering in vision-language models (VLMs) by incorporating an explicit alignment constraint to jointly factorize vision and language activations. The method was applied to LLaVA, revealing that additive steering is most effective in mid-to-late layers, while suppression effects are consistent across all layers. These findings indicate that aligned sparse representations can facilitate more effective intervention-based analysis of multimodal features compared to traditional approaches.
Layer-dependent asymmetries in steering suggest that effective control of vision-language models hinges on the specific layers targeted for intervention.
Sparse Autoencoders (SAEs) have shown promise for analyzing language models, but applying them to vision-language models (VLMs) often yields representations that are difficult to use as controllable cross-modal steering directions. We introduce the Joint Sparse Autoencoder (JSAE), which uses an explicit alignment constraint to jointly factorize sequence-pooled vision and language activations into shared, interpretable image/caption-level features. Applied to LLaVA, JSAE recovers cross-modal features for recognizable concepts (e.g., food and animals). Through bidirectional interventions (additive steering and suppression), we observe a layer-dependent asymmetry under our protocol: additive steering peaks at mid-to-late (pre-output) layers and weakens at both ends, whereas suppression scores remain within a comparable range across all probed layers within statistical noise. Experiments on three VLMs, namely LLaVA-v1.6-Mistral-7B, Llama3-LLaVA-8B, and the MoE-based Qwen3-VL-30B, show related layer-localized effects across architectures. Together, these results suggest that explicitly aligned sparse representations support more controllable intervention-based analysis of multimodal features, within an identifiable layer range, than the unconstrained alternatives tested here.
