Search papers, labs, and topics across Lattice.
NVIDIAAffiliation:Chung-AngDnotitia IncKyungpook National UniversitySamsung Changwon HospitalSNUSoonchunhyang UniversityAug 31, 2026arXiv:2608.30725This study investigates the performance gap in multilingual vision-language encoders between high-resource languages (HRL) and low-resource languages (LRL), revealing that the alignment issue is not due to a linear language classifier bias but rather the divergence of the end-of-sequence (EOS) hidden state's trajectory with depth. By substituting the EOS with its English counterpart, the authors achieve a significant improvement in retrieval performance for Swahili, demonstrating that the encoder's architecture plays a crucial role in LRL success. The findings indicate that adjustments to the encoder's forward path can enhance LRL retrieval without compromising HRL performance, suggesting a new direction for model training strategies.
Swahili retrieval performance skyrockets from 22.1% to 69.1% by addressing the encoder's hidden state trajectory rather than merely tweaking the linear classifier.
Recent multilingual vision--language encoders cover hundreds of languages in a single model, yet on two state-of-the-art instances retrieval on low-resource languages (LRL; e.g. Swahili) trails high-resource ones (HRL; e.g. English) by $30^+$\,pp. We ask where in the trained encoder this gap is located. Prior modality-gap and cross-lingual subspace work suggests a linear language direction at the output crowds out alignment-relevant geometry. We falsify this: LEACE drives the linear language classifier from $>99\%$ to near chance and iterated INLP to $37$--$50\%$ while LRL retrieval moves within $\pm 1.5$\,pp and all tier means within $2.2$\,pp, tracking random controls. The linear bias is a \emph{symptom}, not the cause. Instead, the alignment-causal factor lies along the encoder's forward path: the EOS (end-of-sequence) hidden state's per-language trajectory diverges with depth. Substituting the EOS with its parallel English value three blocks before the projector lifts Swahili from $22.1\%$ to $69.1\%$ on one encoder (and reproduces on the other); three controls rule out pooled-position tautology and English specificity. A front-layer trunk that pulls each language's projection toward the parallel-content centroid corroborates the diagnosis at training time, recovering $+9.6$ / $+17.1$\,pp on LRL XM3600 retrieval (1{,}000-image subset), with consistent gains across three further benchmarks while preserving HRL performance.
