Search papers, labs, and topics across Lattice.
This paper introduces MIXAR, a pixel-based autoregressive language model trained on eight languages with diverse scripts to address multilingual generalization challenges in pixel space. MIXAR outperforms previous pixel-based and comparable token-based models on discriminative and generative multilingual tasks, demonstrating strong cross-lingual transfer. Scaling MIXAR to 0.5B parameters further improves generative capabilities and robustness against orthographic attacks.
Pixel-based language models can now handle multiple languages and scripts, achieving strong multilingual performance and robustness without tokenization.
Pixel-based language models are gaining momentum as alternatives to traditional token-based approaches, promising to circumvent tokenization challenges. However, the inherent perceptual diversity across languages poses a significant hurdle for multilingual generalization in pixel space. This paper introduces MIXAR, the first generative pixel-based language model trained on eight different languages utilizing a range of different scripts. We empirically evaluate MIXAR against previous pixel-based models as well as comparable tokenizer-based models, demonstrating substantial performance improvement on discriminative and generative multilingual tasks. Additionally, we show how MIXAR is robust to languages never seen during the training. These results are further strengthened when scaling the model to 0.5B parameters which not only improves its capabilities in generative tasks like LAMBADA but also its robustness when challenged with input perturbations such as orthographic attacks.
