Search papers, labs, and topics across Lattice.
Nemotron 3 Ultra is a 550 billion parameter Mixture-of-Experts Hybrid Mamba-Attention language model that has been pre-trained on 20 trillion text tokens and fine-tuned using a combination of Supervised Fine Tuning, Reinforcement Learning, and Multi-teacher On-Policy Distillation. This model achieves up to six times higher inference throughput compared to existing state-of-the-art LLMs while maintaining competitive accuracy, making it particularly suited for long-running autonomous tasks. By extending the context length to 1 million tokens, Nemotron 3 Ultra significantly enhances agentic reasoning capabilities in AI applications.
Achieving six times the inference throughput of current LLMs while maintaining accuracy, Nemotron 3 Ultra redefines performance benchmarks for agentic reasoning tasks.
We introduce Nemotron 3 Ultra, a 550 billion total and 55 billion active parameter Mixture-of-Experts Hybrid Mamba-Attention language model. We pre-trained Nemotron 3 Ultra on 20 trillion text tokens, then extended the context length to 1M tokens, and post-trained using Supervised Fine Tuning (SFT), Reinforcement Learning (RL), and Multi-teacher On-Policy Distillation (MOPD). Nemotron 3 Ultra is our most capable model yet, employing multiple key technologies - LatentMoE, Multi Token Prediction (MTP), NVFP4 pre-training, multi-environment RLVR, MOPD, and reasoning budget control. Nemotron 3 Ultra achieves up to ~6x higher inference throughput as compared to state-of-the-art publicly available LLMs while attaining on-par accuracy. The state-of-the-art accuracy, high inference throughput, and 1M token context length make Nemotron 3 Ultra ideal for long-running autonomous agentic tasks. We open-source the base, post-trained, and quantized checkpoints, along with the training data and recipe on HuggingFace.
AI2
NVIDIA