Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
To determine how architectural differences alter latent representation geometry, the authors perform a multi-scale analysis comparing Transformers, State-Space Models (SSMs like Mamba), and hybrid networks across dimensionality, compressibility, and manifold alignment. This investigation addresses whether alternative sub-quadratic architectures learn fundamentally distinct functional representations or merely reach similar behavioral outputs via different internal paths. The results reveal a striking duality: while attention layers skew transformer representations along a single dominant principal direction compared to the isotropic distribution of SSMs, both paradigms exhibit nearly identical effective capacities, concept subspace dimensionalities, and aligned local semantic manifolds.
Transformers collapse their latent space into a single dominant direction while SSMs spread information isotropically, yet both converge on identical effective capacity and local semantic manifolds.
Recent state-space models (SSMs) such as Mamba achieve language modeling performance comparable to transformers despite relying on fundamentally different architectures. This raises an important question: how do these structural differences influence the geometry and functional nature of their internal representations? We study this question through a multi-scale analysis of representations in transformers, SSMs, and hybrid architecture. First, we find that SSMs distribute their representational information evenly across all dimensions, whereas transformer representations are heavily dominated by a single principal direction. By evaluating hybrid architectures, we observe that the representation space becomes increasingly skewed toward a single dominant direction after each attention layer. Next, we explore how the different geometric spread of representations impacts representational capacity through compressibility. Surprisingly, we find that despite their contrasting geometric structures, both architectures exhibit tightly matched effective capacities. We further investigate whether this skewed geometry affects how concepts are encoded. Using rank-constrained probes, we demonstrate that both architectures encode concepts in subspaces of surprisingly similar dimensionality. Furthermore, we demonstrate that the transformers' dominant principal direction does not inherently encode more conceptual information. Finally, we zoom in and examine the alignment between manifolds, either by analyzing representations of specific topics or by looking at the nearest neighborhoods of tokens, and find that they are highly aligned. Ultimately, our analysis suggests that while transformers and SSMs induce different usage of latent space, they display a striking functional convergence at the level of local semantic manifolds.