Search papers, labs, and topics across Lattice.
This paper investigates State Space Models (SSMs) as vision backbones for VLMs, comparing them to Transformer-based encoders in a controlled setting. They find that SSM backbones achieve strong overall performance in VQA and grounding/localization tasks, especially when initialized with matched ImageNet-1K weights. Furthermore, the study reveals that dense-task tuning improves performance across both SSM and ViT backbones, with SSMs remaining competitive at a smaller model scale, suggesting they are a viable alternative to Transformers.
State Space Models can outperform Vision Transformers as vision encoders in VLMs, particularly when model size is a constraint.
Large vision--language models (VLMs) often use a frozen vision backbone, whose image features are mapped into a large language model through a lightweight connector. While transformer-based encoders are the standard visual backbone, we ask whether state space model (SSM) vision backbones can be a strong alternative. We systematically evaluate SSM vision backbones for VLMs in a controlled setting. Under matched ImageNet-1K initialization, the SSM backbone achieves the strongest overall performance across both VQA and grounding/localization. We further adapt both SSM and ViT-family backbones with detection or segmentation training and find that dense-task tuning generally improves performance across families; after this adaptation, the SSM backbone remains competitive while operating at a substantially smaller model scale. We further observe that (i) higher ImageNet accuracy or larger backbones do not reliably translate into better VLM performance, and (ii) some visual backbones are unstable in localization. Based on these findings, we propose stabilization strategies that improve robustness for both backbone families and highlight SSM backbones as a strong alternative to transformer-based vision encoders in VLMs.
