Search papers, labs, and topics across Lattice.
The authors introduce Llama-3-Nanda-10B-Chat (Nanda), a 10B parameter instruction-tuned generative LLM for Hindi, built upon Llama-3-8B with expanded transformer blocks and the Llama Pro methodology. They addressed the challenge of limited Hindi data by employing rigorous data curation, augmentation, and strategic bilingual training to balance Hindi and English corpora. Nanda achieves state-of-the-art performance among open-source Hindi and multilingual models of similar scale, demonstrating significant advantages over existing models.
A new open-source Hindi LLM, Nanda, outperforms existing models of similar scale by strategically balancing Hindi and English training data.
Developing high-quality large language models (LLMs) for moderately resourced languages presents unique challenges in data availability, model adaptation, and evaluation. We introduce Llama-3-Nanda-10B-Chat, or Nanda for short, a state-of-the-art Hindi-centric instruction-tuned generative LLM, designed to push the boundaries of open-source Hindi language models. Built upon Llama-3-8B, Nanda incorporates continuous pre-training with expanded transformer blocks, leveraging the Llama Pro methodology. A key challenge was the limited availability of high-quality Hindi text data; we addressed this through rigorous data curation, augmentation, and strategic bilingual training, balancing Hindi and English corpora to optimize cross-linguistic knowledge transfer. With 10 billion parameters, Nanda stands among the top-performing open-source Hindi and multilingual models of similar scale, demonstrating significant advantages over many existing models. We provide an in-depth discussion of training strategies, fine-tuning techniques, safety alignment, and evaluation metrics, demonstrating how these approaches enabled Nanda to achieve state-of-the-art results. By open-sourcing Nanda, we aim to advance research in Hindi LLMs and support a wide range of real-world applications across academia, industry, and public services.
CMU ML