Search papers, labs, and topics across Lattice.
This paper introduces iLLaDA, an 8B masked diffusion language model that utilizes fully bidirectional attention and is trained from scratch with a masked diffusion objective. By scaling pre-training to 12 trillion tokens and fine-tuning on a 25 billion-token instruction corpus, iLLaDA achieves significant performance improvements across various benchmarks, including a 21.6-point increase on BBH and a 14.5-point increase on MATH. The findings suggest that non-autoregressive training can yield competitive results, challenging the dominance of traditional autoregressive models in the language modeling landscape.
iLLaDA's fully bidirectional diffusion training outperforms traditional autoregressive models, achieving remarkable gains across key language benchmarks.
Modern large language models are predominantly trained with autoregressive factorization and causal attention. We present iLLaDA, an 8B masked diffusion language model trained from scratch with fully bidirectional attention. iLLaDA keeps the masked diffusion objective throughout pre-training and supervised fine-tuning (SFT), scaling pre-training to 12T tokens and fine-tuning on a 25B-token instruction corpus for 12 epochs. We further use variable-length generation for efficiency and introduce confidence-based scoring for multiple-choice evaluation. Compared with LLaDA, iLLaDA improves broadly across general, mathematical, and code benchmarks; for example, iLLaDA-Base improves by 21.6 points on BBH and 14.9 points on ARC-Challenge, while iLLaDA-Instruct improves by 14.5 points on MATH and 16.5 points on HumanEval. Despite its non-autoregressive training, iLLaDA also remains competitive with Qwen2.5 7B on several benchmarks. These results show that fully bidirectional diffusion training from scratch is a competitive path toward strong language models. Model weights and codes: https://github.com/ML-GSAI/LLaDA.
