Search papers, labs, and topics across Lattice.
This study investigates the internal representations of multilingual models by comparing a bilingual pre-trained model with an English-only counterpart across eight languages. The authors find that while token embeddings appear similar post-alignment, deeper hidden states used for predictions reveal significant discrepancies, particularly in the middle transformer layers. This suggests that the assumption of comparability in shared-language representations may be flawed, impacting the interpretation of multilingual model performance in various applications.
Bilingual pretraining can create hidden-state mismatches that mislead researchers about the comparability of multilingual models.
When researchers compare multilingual models for probing, interpretability, or cross-lingual transfer, they often align embedding spaces and assume that shared-language representations are comparable. We show that this assumption can be premature for decoder-only models. We pretrain paired 310M-parameter models (one English-only, one bilingual) across eight typologically diverse languages, separately controlling for English exposure, total compute, and document overlap. After aligning on shared English vocabulary, we test held-out words and find that token embeddings look similar after alignment, but the deeper hidden states that the model uses for prediction do not. This gap holds for all eight languages and survives controls for document overlap and alternative alignment methods. This hidden-state mismatch grows through middle transformer layers, suggesting that it arises from contextual processing rather than the input representations where alignment is performed. Embedding alignment can mask real differences in how models internally represent a shared language, which matters for any downstream study that treats aligned models as interchangeable.
