Search papers, labs, and topics across Lattice.
This paper investigates whether multilingual embeddings encode language-general proficiency by training linear and non-linear probes on Qwen3-Embedding models to predict CEFR levels. While probes perform well in-distribution, achieving $QWK\approx0.7$, performance collapses in cross-corpus evaluations. Residual analysis indicates that probes learn corpus-specific distributional properties rather than a transferable proficiency dimension, suggesting limitations in the language-general proficiency encoded by current multilingual embeddings.
Multilingual embeddings that seem to encode language proficiency in one dataset completely fail to generalize to others, suggesting they're picking up dataset-specific artifacts, not true proficiency.
Do multilingual embedding models encode a language-general representation of proficiency? We investigate this by training linear and non-linear probes on hidden-state activations from Qwen3-Embedding (0.6B, 4B, 8B) to predict CEFR proficiency levels from learner texts across nine corpora and seven languages. We compare five probing architectures against a baseline trained on surface-level text features. Under in-distribution evaluation, probes achieve strong performance ($QWK\approx0.7$), substantially outperforming the surface baseline, with middle layers consistently yielding the best predictions. However, in cross-corpus evaluation performance collapses across all probe types and model sizes. Residual analysis reveals that out-of-distribution probes converge towards predicting uniformly distributed labels, indicating that the learned mappings capture corpus-specific distributional properties (topic, language, task type, rating methodology) rather than an abstract, transferable proficiency dimension. These results suggest that current multilingual embeddings do not straightforwardly encode language-general proficiency, with implications for representation-based approaches to proficiency-adaptive language technology.
