Search papers, labs, and topics across Lattice.
This study employs contrastive-difference CKA (CKA_Delta) to investigate the structural alignment of high-level concepts across various language model architectures, revealing a nuanced relationship between geometric convergence and functional transfer. The findings indicate that while moderate geometric convergence exists, it coexists with near-perfect functional transfer across multiple concept domains, highlighting the limitations of standard CKA in capturing concept-specific convergence. Notably, the results suggest that universality in concept representation may strengthen with larger models, underscoring the need for further exploration with models exceeding 70 billion parameters.
Moderate geometric convergence in LLMs can coexist with near-perfect functional transfer, challenging assumptions about structural alignment across architectures.
Do different LLM architectures encode high-level concepts in structurally compatible ways? We systematically characterize a geometric-functional universality dissociation: across multiple concept domains and architectural families, moderate geometric convergence coexists with near-perfect functional transfer. Using contrastive-difference CKA (CKA_Delta), a training-free diagnostic that computes kernel alignment on per-sample contrastive differences, we isolate concept-specific convergence from generic similarity -- achieving significant discrimination where standard CKA cannot. The dissociation replicates across all six concept domains we test (five with p <= 0.017 geometric discrimination and safety as a converging-functional trend, p = 0.08), including two non-instruction concepts (code-vs-NL, reasoning-vs-recall) validated without system prompts; a single 70B--70B pair provides an observational note that universality may strengthen with scale, requiring replication with additional >=70B models. We position CKA_Delta as a practical regime classifier and architectural outlier detector (Gemma: d = 1.08, AUC = 0.79) rather than an absolute transfer-accuracy predictor, providing a training-free diagnostic for cross-architecture concept monitoring.
