Search papers, labs, and topics across Lattice.
This study investigates the representation differences between Vision Mamba (VMamba) and MambaOut models in high-resolution vision tasks, revealing that VMamba's final stage blocks encode visual information distinctly from MambaOut. Through cross model centered kernel alignment (CKA) analysis, it was found that VMamba produces high-norm tokens primarily in background regions, while MambaOut focuses on foreground tokens, leading to different encoding strategies. The findings demonstrate that VMamba's superior performance in dense prediction tasks is attributed to its organization of semantic evidence across token magnitude and direction, rather than solely the SSM mechanism.
VMamba's unique encoding strategy enables it to outperform MambaOut in high-resolution tasks by leveraging the organization of semantic evidence across token magnitude and direction.
Vision Mamba models replace quadratic self-attention with linear complexity selective state space models (SSMs), emerging as efficient visual backbones. However, MambaOut demonstrates that a Gated CNN block can match or exceed VMamba on image classification, questioning the necessity of SSMs for vision. This raises a fundamental question: do VMamba and MambaOut encode visual information differently at the representation level? To investigate, we apply cross model centered kernel alignment (CKA) analysis and find that VMamba's final stage blocks form representations distinctly different from both MambaOut and its own preceding blocks. We therefore focus on the final block features, decomposing each spatial token into magnitude and direction. MambaOut concentrates class-discriminative information in high-norm foreground tokens that align with Grad-CAM attribution. VMamba, by contrast, produces high-norm tokens predominantly in background regions, misaligned with Grad-CAM, yet preserves discriminative signals primarily in token directions. These observations reveal that the two models rely on different encoding strategies. We connect this difference to high-resolution classification and semantic segmentation. VMamba distributes logit support broadly across object regions, whereas MambaOut relies on sparse dominant tokens, a strategy that becomes less stable as token counts grow. Under full fine-tuning for segmentation, VMamba consistently outperforms MambaOut. These results suggest that VMamba's advantage in dense prediction stems not merely from the SSM mechanism or sequence length, but from how semantic evidence is organized across token magnitude, direction. Ultimately, we conclude that token magnitude and directional structure serve as critical axes for improving visual backbones, particularly under dense supervision.
