Search papers, labs, and topics across Lattice.
This study empirically evaluates the effectiveness of various LoRA (Low-Rank Adaptation) techniques in the context of multilingual instruction tuning. By comparing basic LoRA with four other variants across two datasets and multiple languages, the authors found no significant performance advantage for the more complex LoRA variants in terms of cross-lingual transfer and knowledge retention. Additionally, analysis of hidden embeddings indicated that the layer-wise language representation remains largely consistent across different LoRA techniques, challenging the assumption that architectural complexity enhances multilingual adaptation.
More complex LoRA variants offer no significant benefits over basic LoRA for multilingual instruction tuning, raising questions about the necessity of architectural complexity in this context.
We investigate whether commonly available LoRA variants have an advantage over basic LoRA in multilingual instruction tuning. Experiments involving LoRA and four other variants on two datasets across diverse target languages show that there is no significant advantage in using more complex LoRA variants instead of basic LoRA, with respect to balancing cross-lingual transfer and knowledge retention. An analysis of hidden embeddings reveal that layer-wise language representation remains largely similar across LLMs fine-tuned with different LoRA techniques, suggesting that architectural novelty of LoRA techniques may not translate into better cross-lingual adaptation.
